{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"01_TF_Linear_Regression","provenance":[{"file_id":"https://github.com/madewithml/lessons/blob/master/notebooks/02_Basics/04_Linear_Regression.ipynb","timestamp":1582568936460}],"collapsed_sections":[],"toc_visible":true},"kernelspec":{"name":"python3","display_name":"Python 3"}},"cells":[{"cell_type":"markdown","metadata":{"id":"7nDdR3N2R2PJ","colab_type":"text"},"source":["# Linear Regression\n","\n","In this lesson we will learn about linear regression. We will understand the basic math behind it, implement it in just NumPy and then in native [TensorFlow](https://www.tensorflow.org/install) and then with Keras."]},{"cell_type":"markdown","metadata":{"id":"Izx6GIn_SETR","colab_type":"text"},"source":["<div align=\"left\">\n","<a href=\"https://github.com/madewithml/lessons/blob/master/notebooks/02_Basics/01_Linear_Regression/01_TF_Linear_Regression.ipynb\" role=\"button\"><img class=\"notebook-badge-image\" src=\"https://img.shields.io/static/v1?label=&amp;message=View%20On%20GitHub&amp;color=586069&amp;logo=github&amp;labelColor=2f363d\"></a>&nbsp;\n","<a href=\"https://colab.research.google.com/github/madewithml/lessons/blob/master/notebooks/02_Basics/01_Linear_Regression/01_TF_Linear_Regression.ipynb\"><img class=\"notebook-badge-image\" src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"></a>\n","</div>"]},{"cell_type":"markdown","metadata":{"id":"NGE2LxIQuSRa","colab_type":"text"},"source":["# Overview"]},{"cell_type":"markdown","metadata":{"id":"Hi-KdxD_uSln","colab_type":"text"},"source":["Our goal is to learn a linear model $\\hat{y}$ that models $y$ given $X$. \n","\n","$\\hat{y} = XW + b$\n","* $\\hat{y}$ = predictions | $\\in \\mathbb{R}^{NX1}$ ($N$ is the number of samples)\n","* $X$ = inputs | $\\in \\mathbb{R}^{NXD}$ ($D$ is the number of features)\n","* $W$ = weights | $\\in \\mathbb{R}^{DX1}$ \n","* $b$ = bias | $\\in \\mathbb{R}^{1}$ "]},{"cell_type":"markdown","metadata":{"id":"DxOri3K3uSsB","colab_type":"text"},"source":["* **Objective:**  Use inputs $X$ to predict the output $\\hat{y}$ using a linear model. The model will be a line of best fit that minimizes the distance between the predicted (model's output) and target (ground truth) values. Training data $(X, y)$ is used to train the model and learn the weights $W$ using gradient descent.\n","* **Advantages:**\n","  * Computationally simple.\n","  * Highly interpretable.\n","  * Can account for continuous and categorical features.\n","* **Disadvantages:**\n","  * The model will perform well only when the data is linearly separable (for classification).\n","  * Usually not used for classification and only for regression.\n","* **Miscellaneous:** You can also use linear regression for binary classification tasks where if the predicted continuous value is above a threshold, it belongs to a certain class. But we will cover better techniques for classification in future lessons and will focus on linear regression for continuous regression tasks only.\n"]},{"cell_type":"markdown","metadata":{"id":"G8EkjWiKd68P","colab_type":"text"},"source":["# Generate data"]},{"cell_type":"markdown","metadata":{"id":"2u0wnvwWgjmT","colab_type":"text"},"source":["We're going to create some simple dummy data to apply linear regression on. It's going to create roughly linear data (`y = 3.5X + noise`); the random noise is added to create realistic data that doesn't perfectly align in a line. Our goal is to have the model converge to a similar linear equation (there will be slight variance since we added some noise)."]},{"cell_type":"code","metadata":{"id":"a39zl86Fg1CL","colab_type":"code","colab":{}},"source":["import numpy as np\n","import pandas as pd\n","import matplotlib.pyplot as plt"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"9rUjxlhvhzIR","colab_type":"code","colab":{}},"source":["SEED = 1234\n","NUM_SAMPLES = 50"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"OOT0wOiZX9eJ","colab_type":"code","colab":{}},"source":["# Set seed for reproducibility\n","np.random.seed(SEED)"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"vCcF_QkYdozl","colab_type":"code","colab":{}},"source":["# Generate synthetic data\n","def generate_data(num_samples):\n","    \"\"\"Generate dummy data for linear regression.\"\"\"\n","    X = np.array(range(num_samples))\n","    random_noise = np.random.uniform(-10,20,size=num_samples)\n","    y = 3.5*X + random_noise # add some noise\n","    return X, y"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"kzSANn29g36a","colab_type":"code","outputId":"673ac2b9-ce37-48e1-f96f-49b5b0ae9b3d","executionInfo":{"status":"ok","timestamp":1583941044634,"user_tz":420,"elapsed":1272,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":102}},"source":["# Generate random (linear) data\n","X, y = generate_data(num_samples=NUM_SAMPLES)\n","data = np.vstack([X, y]).T\n","print (data[:5])"],"execution_count":95,"outputs":[{"output_type":"stream","text":["[[ 0.         -4.25441649]\n"," [ 1.         12.16326313]\n"," [ 2.         10.13183217]\n"," [ 3.         24.06075751]\n"," [ 4.         27.39927424]]\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"4ckWsUsKhCDC","colab_type":"code","outputId":"241ced6f-f885-4e1e-8944-eecea65e5367","executionInfo":{"status":"ok","timestamp":1583941044634,"user_tz":420,"elapsed":816,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":204}},"source":["# Load into a Pandas DataFrame\n","df = pd.DataFrame(data, columns=['X', 'y'])\n","X = df[['X']].values\n","y = df[['y']].values\n","df.head()"],"execution_count":96,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>X</th>\n","      <th>y</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>0.0</td>\n","      <td>-4.254416</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>1.0</td>\n","      <td>12.163263</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>2.0</td>\n","      <td>10.131832</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>3.0</td>\n","      <td>24.060758</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>4.0</td>\n","      <td>27.399274</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["     X          y\n","0  0.0  -4.254416\n","1  1.0  12.163263\n","2  2.0  10.131832\n","3  3.0  24.060758\n","4  4.0  27.399274"]},"metadata":{"tags":[]},"execution_count":96}]},{"cell_type":"code","metadata":{"id":"qh3z65QXhCj5","colab_type":"code","outputId":"0623b0a1-d729-4784-9238-d2f2471c92b3","executionInfo":{"status":"ok","timestamp":1583941045727,"user_tz":420,"elapsed":1424,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":281}},"source":["# Scatter plot\n","plt.title(\"Generated data\")\n","plt.scatter(x=df['X'], y=df['y'])\n","plt.show()"],"execution_count":97,"outputs":[{"output_type":"display_data","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAXcAAAEICAYAAACktLTqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAbGElEQVR4nO3de5RdZXnH8e/PSdAR0OEyxjAQEzSE\ngtSkTlEbtYgXorUSWS6EqsVKG2+spRXRoF2KdlHSUqV2YbGhUrBLEcolspClpoSKtaBOTMpFoAaE\nkjEkwyUFJY0kPP3j7IHN5JyZc/bZ57L3/n3WOmvOeffe57w7DM955708ryICMzMrl2f1ugJmZpY/\nB3czsxJycDczKyEHdzOzEnJwNzMrIQd3M7MScnA36wBJ75X0Hy2cf6+kN3SyTlYtDu7WNZJOkvQj\nSb+WtC15/iFJ6nXdppL075L+tNf1qEdSSHpJr+th/c3B3bpC0unAl4BzgRcCc4APAEuBvbpcl1nd\n/DyzXnBwt46T9Hzg88CHIuKKiHgsajZExLsiYmdy3rMl/a2k/5G0VdJXJA0mx46RtFnS6Umrf4uk\nP0l9RjPXflLSA8A/S9pP0rWSJiQ9kjw/ODn/bOA1wPmSfiXp/KT8cElrJT0s6S5JJ6Y+/wBJ10h6\nVNKPgRfP8G/yHkn3SXpI0qenHDta0k2Stif3eb6kvZJjNyan/VdSt3dOdy9WXQ7u1g2vAp4NfGuG\n81YBhwGLgZcAI8BnUsdfCDw/KT8V+LKk/Vq4dn/gRcAKar/7/5y8ngfsAM4HiIhPAz8ATouIfSLi\nNEl7A2uBbwAvAE4C/kHSEcn7fxn4P2Au8L7kUVdyzQXAe4CDgAOAdDDeDfw5cCC1f7vXAx9K6vba\n5JyXJXW7bLp7sQqLCD/86OgDeDfwwJSy/wS2UwtErwUE/Bp4ceqcVwG/SJ4fk5w7K3V8G/DKJq/9\nDfCcaeq4GHgk9frfgT9NvX4n8IMp1/wj8FlgAHgCODx17K+A/2jwWZ8Bvpl6vXdSvzc0OP+jwNWp\n1wG8pNl78aOaD/c9Wjc8BBwoaVZE7AKIiN8DkLSZWstzGHgusD41vipqgfOp95m8PvE4sE+T105E\nxP89dVB6LnAesAyYbP3vK2kgInbXuYcXAa+QtD1VNgv4l+TzZwH3p47dV/+fAqi11p86NyJ+Lemh\nVN0OA74IjCb3NQtY3+jNMtyLVYC7ZawbbgJ2AsdPc86D1FrmR0bEUPJ4fkTs08T7N3Pt1PSnpwOL\ngFdExPOo/fUAtS+FeuffD3w/9f5DUesW+SAwAewCDkmdP2+a+m5Jn5sE5wNSxy8A7gQWJnX7VKpe\n9cx0L1ZBDu7WcRGxHfgctT7qd0jaV9KzJC2m1iVBRDwJXAicJ+kFAJJGJB3XxPtnuXZfal8I2yXt\nT617JW0rcGjq9bXAYclA6Ozk8buSfitpHV8FnCXpuUmf+inTfPYVwFslvToZKP08z/x/cV/gUeBX\nkg4HPjhD3Wa6F6sgB3frioj4G+BjwCeoBaet1PqsP0mt/53k+SbgZkmPAv9GrUXajFav/TtgkFqr\n/2bgO1OOfwl4RzL75O8j4jHgTdQGUn8JPAD8NbWBYoDTqHURPQBcTG2As66IuB34MLXB2S3AI8Dm\n1CkfB/4IeIzal9ZlU97iLOCSZDbNiU3ci1WQIrxZh5lZ2bjlbmZWQg7uZmYl5OBuZlZCDu5mZiXU\nF4uYDjzwwJg/f36vq2FmVijr169/MCKG6x3ri+A+f/58xsbGel0NM7NCkdRwJbS7ZczMSsjB3cys\nhBzczcxKyMHdzKyEHNzNzEqoL2bLmJlVzZoN45z73bv45fYdHDQ0yBnHLWL5kpHc3t/B3cysy9Zs\nGOfMq25lxxO1vVTGt+/gzKtuBcgtwM/YLSPpomRD4ttSZZdJ2pg87pW0MSmfL2lH6thXcqmlmVmJ\nnPvdu54K7JN2PLGbc797V26f0UzL/WJqm+1+bbIgIt45+VzSF4D/TZ1/d0QszquCZmZl88vtO1oq\nz2LGlntE3Ag8XO+YahtWnghcmluNzMxK7qChwZbKs2h3tsxrgK0R8fNU2QJJGyR9X9JrGl0oaYWk\nMUljExMTbVbDzKw4zjhuEYOzB55RNjh7gDOOa3bjsZm1O6B6Ms9stW8B5kXEQ5JeDqyRdGREPDr1\nwohYDawGGB0d9XZQZlZYrc58mTzWl7NlJM0CTgBePlkWETup7XJPRKyXdDdwGOCsYGZWSllnvixf\nMpJrMJ+qnW6ZNwB3RsRTG/tKGpY0kDw/FFgI3NNeFc3M+lc3Zr5k0cxUyEuBm4BFkjZLOjU5dBJ7\nDqS+FrglmRp5BfCBiKg7GGtmVgbdmPmSxYzdMhFxcoPy99YpuxK4sv1qmZkVw0FDg4zXCeSTM186\nvRK1EeeWMTNrw3QzXyb748e37yB4uj9+zYbxjtfLwd3MrA3Ll4xwzglHMTI0iICRoUHOOeEoli8Z\n6Wl/vHPLmJm1qdHMl172x7vlbmbWId1YidqIg7uZ2RRrNoyzdNU6Fqz8NktXrcvcR96NlaiNuFvG\nzCqr3kwWILd0vN1YidqIInq/8n90dDTGxryI1cy6Z+rKUqi1qp8z+1k88vgTe5w/MjTID1ce280q\nzkjS+ogYrXfMLXczK716LfRGM1mmlk3q9aKkVjm4m1mpNcr90iiIN9KNQdA8eUDVzEqtUQt9QKp7\n/tDg7J4NgubJLXczK7VG3Sm7IxicPbBHn/tZbzsS6M0gaJ4c3M2s1BrlfhlJ9b3XC+JFC+ZTObib\nWamdcdyiurNiJgN50YN4Iw7uZlZqvZxr3ksO7mbWU91IiVvmFnojDu5m1jNZt6izmTm4m1nPTJcS\ntx+De6823sjCwd3MeqZft6irp2h/ZTSzh+pFkrZJui1VdpakcUkbk8dbUsfOlLRJ0l2SjutUxc2s\n+HqZErdV/boRdiPNrFC9GFhWp/y8iFicPK4DkHQEtY2zj0yu+QdJA3WuNTPraUrcVhXprwxoIrhH\nxI3Aw02+3/HANyNiZ0T8AtgEHN1G/cysxKbboq7fFOmvDGivz/00SX8MjAGnR8QjwAhwc+qczUnZ\nHiStAFYAzJs3r41qmFm/yDLgWJRpitMthupHWROHXQC8GFgMbAG+0OobRMTqiBiNiNHh4eGM1TCz\nfjE54Di+fQfB0wOOWXcx6jdF+isDMrbcI2Lr5HNJFwLXJi/HgUNSpx6clJlZyRVtWmMWRfkrAzK2\n3CXNTb18OzA5k+Ya4CRJz5a0AFgI/Li9KppZERRtwLHsZmy5S7oUOAY4UNJm4LPAMZIWAwHcC7wf\nICJul3Q58DNgF/DhiGgtI76ZFVKj7Iv9OuBYdjMG94g4uU7xV6c5/2zg7HYqZWbFU7QBx7LzClUz\ny0VVsy/2Kwd3M8tNkQYcy857qJqZlZBb7mZWKEXKzNhLDu5mVhjTZWYE9/enObibWWE0Wih11jW3\ns3PXk4VJx9sNDu5m1hV5dKc0WhC1fccTe5SVbXVsqzygamYdl1femVYXRFV5dayDu5l1XF4bXTTK\n/77fc2fXPb/Kq2PdLWNmDeU1MyWvvDONFkoBXh07hYO7mdWV556heeadmW6hlGfLPM3B3czqyjOF\nb5a8M63+1eDVsc/k4G5mdeWZwrfVvDN5/tVQVQ7uZla3lZx3Ct9WWtZV2Pij0zxbxqziGk1TfN3h\nw3VnpnRjkNIbf7TPwd2s4hq1km+4c6Jne4Y2+uugylMbW+VuGbOKm66V3KtBSm/80T4Hd7OSaXWW\nST9uj+eNP9rXzB6qFwFvBbZFxEuTsnOBPwR+A9wN/ElEbJc0H7gDmFx2dnNEfKAD9TazOrLMMunX\nVrKnNranmT73i4FlU8rWAi+NiN8G/hs4M3Xs7ohYnDwc2M26KMsy/+VLRnrWt26d08wG2TcmLfJ0\n2fdSL28G3pFvtcwsi6yzTNxKLp88+tzfB1yWer1A0gbgUeAvIuIHOXyGmTWhW/3n3g2p/7U1FVLS\np4FdwNeToi3AvIhYAnwM+Iak5zW4doWkMUljExMT7VTDzBKNsibm2X+eV/pe66zMwV3Se6kNtL4r\nIgIgInZGxEPJ8/XUBlsPq3d9RKyOiNGIGB0eHs5aDbPSW7NhnKWr1rFg5bdZumrdtEG0G/3neaXv\ntc7K1C0jaRnwCeD3I+LxVPkw8HBE7JZ0KLAQuCeXmppVUJbZL53uP/fq0WKYseUu6VLgJmCRpM2S\nTgXOB/YF1kraKOkryemvBW6RtBG4AvhARDzcobqblV4/tpK9erQYmpktc3Kd4q82OPdK4Mp2K2Vm\nNf3YSu7XefH2TM4tY9bH+rGV7HnxxeD0A2Z9rF9byZ4X3/8c3M36mHOsWFYO7mZ9zq1ky8J97mZm\nJeTgbmZWQu6WMSso53ex6Ti4mxVQlpWrVi0O7mYFNN3K1emCu1v71eHgblZAWVauurVfLQ7uZh2Q\npYXcyjVZ8rZnbe1bMXm2jFnOsuQ7b/WaLHnb+zFPjXWOg7tZzrJkcmz1miz5XfoxT411jrtlzHKW\npYWc5ZpWV672a54a6wy33M1ylqWF3I1WtbM5Votb7mY5y9JC7lar2nlqqsPB3SxnWTI5Ovuj5U3J\n3tY9NTo6GmNjY72uhpkX+VihSFofEaP1jrnlbpbwIh8rk6YGVCVdJGmbpNtSZftLWivp58nP/ZJy\nSfp7SZsk3SLpdzpVebM89eNm1GZZNTtb5mJg2ZSylcD1EbEQuD55DfBmYGHyWAFc0H41zTrPi3ys\nTJoK7hFxI/DwlOLjgUuS55cAy1PlX4uam4EhSXPzqKxZJ3mRj5VJO33ucyJiS/L8AWBO8nwEuD91\n3uakbEuqDEkrqLXsmTdvXhvVMMvHdNMR8xpo9YCtdUsuA6oREZJamnYTEauB1VCbLZNHPcza0Wg6\nIpDLQKsHbK2b2gnuWyXNjYgtSbfLtqR8HDgkdd7BSZlZ36u3yGfpqnW5ZFN0VkbrpnbSD1wDnJI8\nPwX4Vqr8j5NZM68E/jfVfWNWOHkNtHrA1rqp2amQlwI3AYskbZZ0KrAKeKOknwNvSF4DXAfcA2wC\nLgQ+lHutzboor4FWD9haNzXVLRMRJzc49Po65wbw4XYqZdZP8hpodVZG6yavUDWbQV4Drc4fY93k\n3DJmGS1dta7uVncjQ4P8cOWxPaiRVc10uWWcz90sIw+QWj9zcDfLyAOk1s8c3M0yyrJJtVm3eEDV\nLCMPkFo/c3A3a4O3rbN+5W4ZM7MScnA3MyshB3czsxJyn7v1Hec8N2ufg7v1lSw5z1v9MvCXh1WB\nu2Wsr7S6SfXkl8H49h0ET38ZrNlQfwuBVs83KyoHd+srrS7pb/XLoNXzzYrKwd36SqtL+lv9MnA+\nGKsKB3frK60u6W/1y8D5YKwqHNytryxfMsI5JxzFyNAgopY+95wTjpp2A4xWvgycD8aqwrNlrCem\nm7HSypL+VvO7OB+MVUXmzTokLQIuSxUdCnwGGAL+DJhIyj8VEddN917erKNapk53hFrreboWupnt\nqSObdUTEXRGxOCIWAy8HHgeuTg6fN3lspsBu1eMZK2adl1ef++uBuyPivpzez0rMM1bMOi+v4H4S\ncGnq9WmSbpF0kaT96l0gaYWkMUljExMT9U6xkvKMFbPOazu4S9oLeBvwr0nRBcCLgcXAFuAL9a6L\niNURMRoRo8PDw+1WwwrEM1bMOi+P2TJvBn4aEVsBJn8CSLoQuDaHz7ASyTpjxTlhzJqXR3A/mVSX\njKS5EbElefl24LYcPsNKptUdjLIkFDOrsraCu6S9gTcC708V/42kxUAA9045ZpbJdDNs8soWaVYm\nbQX3iPg1cMCUsve0VSOzOlqdYeOWvlWd0w9YIbQ6w8Zz6a3qHNytEFqdYeO59FZ1Du5WCK0mFPNc\neqs6Jw6zwmhlhs0Zxy2qm7/Gc+mtKhzcrZSc/dGqzsHdSqvVufRmZeLgbrnwnHKz/uLgbm3znHKz\n/uPgbi2p10LPsnrUzDrLwd2a1qiFPjWwT/KccrPe8Tx3a1qjFvqAVPd8zyk36x233Cus1UHQRi3x\n3REMzh7wnHKzPuKWe0VNdrGMb99B8HQXy5oN4w2vadQSn1wt2uzqUTPrPLfcKyrLIOh0qz49p9ys\nvzi4V1SWxFpe9WlWHA7uFXXQ0CDjdQL5TIOgbqGbFYODewXUGzh1Yi2zcvOAask1GjgFPAhqVmJt\nt9wl3Qs8BuwGdkXEqKT9gcuA+dT2UT0xIh5p97OsddMNnP5w5bHef9SspPLqlnldRDyYer0SuD4i\nVklambz+ZE6fZS3IMnCaZ64Yf0mY9UanumWOBy5Jnl8CLO/Q59gMsuxIlNf+o1nm0ptZPvII7gF8\nT9J6SSuSsjkRsSV5/gAwJ4fPqZQ1G8ZZumodC1Z+m6Wr1mUOiK3uPQr57T/qTarNeiePbplXR8S4\npBcAayXdmT4YESEppl6UfBGsAJg3b14O1Simet0WQG7dIlnmpmedJjmVN6k26522g3tEjCc/t0m6\nGjga2CppbkRskTQX2FbnutXAaoDR0dE9gn8VNOrbfs7sZ+WaQrfVuel5TZPM60vCzFrXVreMpL0l\n7Tv5HHgTcBtwDXBKctopwLfa+ZyyatRt8cjjT9Q9v1st3uVLRnKZJpmlS8jM8tFuy30OcLVqKV9n\nAd+IiO9I+glwuaRTgfuAE9v8nFJqNVh3s8Wbx0pUpysw6522gntE3AO8rE75Q8Dr23nvKmjUbTE0\nOJudu55suVukH6cdOl2BWW94hWoPNeq2OOttR7bcLeJph2aW5twyPTRTt0UrLV7vY2pmaQ7uPZZX\nt4WnHZpZmrtlSiLLSlQzKy8H9y7Ja8VpI552aGZp7pbpgjwTcTXiaYdmlubg3gXdGuz0tEMzm+Tg\n3gV5D3b243x2M+sv7nPvgjwHOz2f3cya4eDeBXkOdjqNrpk1w90yXZDnYKfns5tZMxzcuySvwU6n\n0TWzZrhbpmA8n93MmuGWe8F4PruZNcPBvYA8n93MZuJuGTOzEnLLfRpeLGRmReXg3kA38sGYmXWK\nu2Ua8GIhMyuyzMFd0iGSbpD0M0m3S/pIUn6WpHFJG5PHW/Krbvd4sZCZFVk73TK7gNMj4qeS9gXW\nS1qbHDsvIv62/ep1R72+dS8WMrMiy9xyj4gtEfHT5PljwB1A4TqjGyXiet3hw14sZGaFlUufu6T5\nwBLgR0nRaZJukXSRpP0aXLNC0piksYmJiTyqkUmjvvUb7pzgnBOOYmRoEAEjQ4Occ8JRHkw1s0JQ\nRLT3BtI+wPeBsyPiKklzgAeBAP4SmBsR75vuPUZHR2NsbKytemS1YOW3qfcvIOAXq/6g29UxM2ua\npPURMVrvWFstd0mzgSuBr0fEVQARsTUidkfEk8CFwNHtfEaneWNpMyujdmbLCPgqcEdEfDFVPjd1\n2tuB27JXr/PyTsTV6Y2wzcya0c5smaXAe4BbJW1Myj4FnCxpMbVumXuB97dVww7LMxGXFz6ZWb9o\nu889D73sc8/T0lXr6k6fHBka5Icrj+1BjcyszDrW527P5IVPZtYvnFsmIy98MrN+5pZ7Bl74ZGb9\nzsE9Ay98MrN+526ZDKbrW/cuSWbWD0oZ3Du9yYb71s2s35WuW6ZRf3iei4nyXvhkZpa30rXcZ9pk\nI48WfZ4Ln8zMOqF0i5gaJQKDWus6HfgHZw94wNPMCqtSi5ga9XsPSN42z8wqo3TBvVF/+O4Gf6F4\n9aiZlVHpgvvyJSN155qPOLWvmVVI6QZUgYZzzdMZG8EzXMysvEoZ3OvxDBczq5LKBHdo3KI3Myub\n0vW5m5mZg7uZWSk5uJuZlVDHgrukZZLukrRJ0spOfY6Zme2pI8Fd0gDwZeDNwBHUNs0+ohOfZWZm\ne+pUy/1oYFNE3BMRvwG+CRzfoc8yM7MpOhXcR4D7U683J2VPkbRC0piksYmJiQ5Vw8ysmno2oBoR\nqyNiNCJGh4eHe1UNM7NS6tQipnHgkNTrg5OyXHV6xyUzs6LqVHD/CbBQ0gJqQf0k4I/y/IDJHZcm\nc8VM7rgEOMCbWeV1pFsmInYBpwHfBe4ALo+I2/P8jJl2XDIzq7KO5ZaJiOuA6zr1/o3ysDs/u5lZ\ngVeoNsrD7vzsZmYFDu6NdlxyfnYzswKn/HV+djOzxgob3MH52c3MGilst4yZmTXm4G5mVkIO7mZm\nJeTgbmZWQg7uZmYlpIjodR2QNAHc18ZbHAg8mFN1isT3XS2+72pp5r5fFBF10+r2RXBvl6SxiBjt\ndT26zfddLb7vamn3vt0tY2ZWQg7uZmYlVJbgvrrXFegR33e1+L6rpa37LkWfu5mZPVNZWu5mZpbi\n4G5mVkKFDu6Slkm6S9ImSSt7XZ9OknSRpG2SbkuV7S9praSfJz/362Ud8ybpEEk3SPqZpNslfSQp\nL/t9P0fSjyX9V3Lfn0vKF0j6UfL7fpmkvXpd106QNCBpg6Rrk9dVue97Jd0qaaOksaQs8+96YYO7\npAHgy8CbgSOAkyUd0dtaddTFwLIpZSuB6yNiIXB98rpMdgGnR8QRwCuBDyf/jct+3zuBYyPiZcBi\nYJmkVwJ/DZwXES8BHgFO7WEdO+kj1PZenlSV+wZ4XUQsTs1vz/y7XtjgDhwNbIqIeyLiN8A3geN7\nXKeOiYgbgYenFB8PXJI8vwRY3tVKdVhEbImInybPH6P2P/wI5b/viIhfJS9nJ48AjgWuSMpLd98A\nkg4G/gD4p+S1qMB9TyPz73qRg/sIcH/q9eakrErmRMSW5PkDwJxeVqaTJM0HlgA/ogL3nXRNbAS2\nAWuBu4HtEbErOaWsv+9/B3wCeDJ5fQDVuG+ofYF/T9J6SSuSssy/64XeicmeFhEhqZTzWiXtA1wJ\nfDQiHq015mrKet8RsRtYLGkIuBo4vMdV6jhJbwW2RcR6Scf0uj498OqIGJf0AmCtpDvTB1v9XS9y\ny30cOCT1+uCkrEq2SpoLkPzc1uP65E7SbGqB/esRcVVSXPr7nhQR24EbgFcBQ5ImG2Rl/H1fCrxN\n0r3UulmPBb5E+e8bgIgYT35uo/aFfjRt/K4XObj/BFiYjKTvBZwEXNPjOnXbNcApyfNTgG/1sC65\nS/pbvwrcERFfTB0q+30PJy12JA0Cb6Q23nAD8I7ktNLdd0ScGREHR8R8av8/r4uId1Hy+waQtLek\nfSefA28CbqON3/VCr1CV9BZqfXQDwEURcXaPq9Qxki4FjqGWBnQr8FlgDXA5MI9ayuQTI2LqoGth\nSXo18APgVp7ug/0UtX73Mt/3b1MbPBug1gC7PCI+L+lQai3a/YENwLsjYmfvato5SbfMxyPirVW4\n7+Qer05ezgK+ERFnSzqAjL/rhQ7uZmZWX5G7ZczMrAEHdzOzEnJwNzMrIQd3M7MScnA3MyshB3cz\nsxJycDczK6H/B8KVIz4zdljAAAAAAElFTkSuQmCC\n","text/plain":["<Figure size 432x288 with 1 Axes>"]},"metadata":{"tags":[]}}]},{"cell_type":"markdown","metadata":{"id":"Myxvm0YWlppI","colab_type":"text"},"source":["# NumPy"]},{"cell_type":"markdown","metadata":{"id":"aHhypQvalsTd","colab_type":"text"},"source":["Now that we have our data prepared, we'll first implement linear regression using just NumPy. This will let us really understand the underlying operations."]},{"cell_type":"markdown","metadata":{"id":"kzjHPncuQpuE","colab_type":"text"},"source":["## Split data"]},{"cell_type":"markdown","metadata":{"id":"ElbTBdP8Q6k5","colab_type":"text"},"source":["Since our task is a regression task, we will randomly split our dataset into **three** sets: train, validation and test data splits.\n","\n","* train: used to train our model.\n","* val : used to validate our model's performance during training.\n","* test: used to do an evaluation of our fully trained model."]},{"cell_type":"code","metadata":{"id":"rrTd9wP9Qv2_","colab_type":"code","colab":{}},"source":["TRAIN_SIZE = 0.7\n","VAL_SIZE = 0.15\n","TEST_SIZE = 0.15\n","SHUFFLE = True"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"w3cWO_EwRC72","colab_type":"code","colab":{}},"source":["# Shuffle data\n","if SHUFFLE:\n","    indices = list(range(NUM_SAMPLES))\n","    np.random.shuffle(indices)\n","    X = X[indices]\n","    y = y[indices]"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"mhnoMWg_gZme","colab_type":"text"},"source":["**NOTE**: Be careful not to shuffle X and y separately because then the inputs won't correspond to the outputs!"]},{"cell_type":"code","metadata":{"id":"CfmqoOyIRUop","colab_type":"code","colab":{}},"source":["# Split indices\n","train_start = 0\n","train_end = int(0.7*NUM_SAMPLES)\n","val_start = train_end\n","val_end = int((TRAIN_SIZE+VAL_SIZE)*NUM_SAMPLES)\n","test_start = val_end"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"F8JKSPgXSBf6","colab_type":"code","outputId":"d2d9263c-0df1-459a-e368-20b1e94ce6de","executionInfo":{"status":"ok","timestamp":1583941051045,"user_tz":420,"elapsed":551,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":68}},"source":["# Split data\n","X_train = X[train_start:train_end]\n","y_train = y[train_start:train_end]\n","X_val = X[val_start:val_end]\n","y_val = y[val_start:val_end]\n","X_test = X[test_start:]\n","y_test = y[test_start:]\n","print (f\"X_train: {X_train.shape}, y_train: {y_train.shape}\")\n","print (f\"X_val: {X_val.shape}, y_test: {y_val.shape}\")\n","print (f\"X_test: {X_test.shape}, y_test: {y_test.shape}\")"],"execution_count":101,"outputs":[{"output_type":"stream","text":["X_train: (35, 1), y_train: (35, 1)\n","X_val: (7, 1), y_test: (7, 1)\n","X_test: (8, 1), y_test: (8, 1)\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"ok62daFIQv9x","colab_type":"text"},"source":["## Standardize data"]},{"cell_type":"markdown","metadata":{"id":"0pQweIQuUTpl","colab_type":"text"},"source":["We need to standardize our data (zero mean and unit variance) or our models can optimize quickly when we are training.\n","\n","$z = \\frac{x_i - \\mu}{\\sigma}$\n","* $z$ = standardized value\n","* $x_i$ = inputs\n","* $\\mu$ = mean\n","* $\\sigma$ = standard deviation"]},{"cell_type":"code","metadata":{"id":"pM1h8CrbQyTG","colab_type":"code","colab":{}},"source":["def standardize_data(data, mean, std):\n","    return (data - mean)/std"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"HWvKMt2-Ui3G","colab_type":"code","colab":{}},"source":["# Determine means and stds\n","X_mean = np.mean(X_train)\n","X_std = np.std(X_train)\n","y_mean = np.mean(y_train)\n","y_std = np.std(y_train)"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"XM1EXQFeU3cL","colab_type":"text"},"source":["We need to treat the validation and test sets as if they were hidden datasets. So we only use the train set to determine the mean and std to avoid biasing our training process."]},{"cell_type":"code","metadata":{"id":"csHkTwpyUvt4","colab_type":"code","colab":{}},"source":["# Standardize\n","X_train = standardize_data(X_train, X_mean, X_std)\n","y_train = standardize_data(y_train, y_mean, y_std)\n","X_val = standardize_data(X_val, X_mean, X_std)\n","y_val = standardize_data(y_val, y_mean, y_std)\n","X_test = standardize_data(X_test, X_mean, X_std)\n","y_test = standardize_data(y_test, y_mean, y_std)"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"qKkTM0n1V8SN","colab_type":"code","outputId":"9c23d591-13cb-4746-c99c-e618d9596425","executionInfo":{"status":"ok","timestamp":1583941054197,"user_tz":420,"elapsed":433,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":119}},"source":["# Check (means should be ~0 and std should be ~1)\n","print (f\"X_train: mean: {np.mean(X_train, axis=0)[0]:.1f}, std: {np.std(X_train, axis=0)[0]:.1f}\")\n","print (f\"y_train: mean: {np.mean(y_train, axis=0)[0]:.1f}, std: {np.std(y_train, axis=0)[0]:.1f}\")\n","print (f\"X_val: mean: {np.mean(X_val, axis=0)[0]:.1f}, std: {np.std(X_val, axis=0)[0]:.1f}\")\n","print (f\"y_val: mean: {np.mean(y_val, axis=0)[0]:.1f}, std: {np.std(y_val, axis=0)[0]:.1f}\")\n","print (f\"X_test: mean: {np.mean(X_test, axis=0)[0]:.1f}, std: {np.std(X_test, axis=0)[0]:.1f}\")\n","print (f\"y_test: mean: {np.mean(y_test, axis=0)[0]:.1f}, std: {np.std(y_test, axis=0)[0]:.1f}\")"],"execution_count":105,"outputs":[{"output_type":"stream","text":["X_train: mean: -0.0, std: 1.0\n","y_train: mean: 0.0, std: 1.0\n","X_val: mean: -0.5, std: 0.5\n","y_val: mean: -0.6, std: 0.5\n","X_test: mean: -0.6, std: 0.9\n","y_test: mean: -0.6, std: 0.9\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"jyqSl8jyQzKz","colab_type":"text"},"source":["## Weights"]},{"cell_type":"markdown","metadata":{"id":"3xQZ0Hbln6TR","colab_type":"text"},"source":["Our goal is to learn a linear model $\\hat{y}$ that models $y$ given $X$. \n","\n","$\\hat{y} = XW + b$\n","* $\\hat{y}$ = predictions | $\\in \\mathbb{R}^{NX1}$ ($N$ is the number of samples)\n","* $X$ = inputs | $\\in \\mathbb{R}^{NXD}$ ($D$ is the number of features)\n","* $W$ = weights | $\\in \\mathbb{R}^{DX1}$ \n","* $b$ = bias | $\\in \\mathbb{R}^{1}$ "]},{"cell_type":"markdown","metadata":{"id":"Baph5j7xn9XE","colab_type":"text"},"source":["1. Randomly initialize the model's weights $W$."]},{"cell_type":"code","metadata":{"id":"aAOAboM0oFbV","colab_type":"code","colab":{}},"source":["INPUT_DIM = X_train.shape[1] # X is 1-dimensional\n","OUTPUT_DIM = y_train.shape[1] # y is 1-dimensional"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"chU5n3IgkVci","colab_type":"code","outputId":"b92e6078-5f46-44d0-8f5b-8609285c5f71","executionInfo":{"status":"ok","timestamp":1583941057187,"user_tz":420,"elapsed":582,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":51}},"source":["# Initialize random weights\n","W = 0.01 * np.random.randn(INPUT_DIM, OUTPUT_DIM)\n","b = np.zeros((1, 1))\n","print (f\"W: {W.shape}\")\n","print (f\"b: {b.shape}\")"],"execution_count":107,"outputs":[{"output_type":"stream","text":["W: (1, 1)\n","b: (1, 1)\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"_QjDXSXfWv7Y","colab_type":"text"},"source":["## Model"]},{"cell_type":"markdown","metadata":{"id":"tOqP0jrBoMMd","colab_type":"text"},"source":["2. Feed inputs $X$ into the model to receive the predictions $\\hat{y}$.\n","  * $\\hat{y} = XW + b$"]},{"cell_type":"code","metadata":{"id":"-TBnfN4mkVa6","colab_type":"code","outputId":"a9cfb411-3832-4837-ec3d-4da3de82e86b","executionInfo":{"status":"ok","timestamp":1583941058638,"user_tz":420,"elapsed":413,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["# Forward pass [NX1] · [1X1] = [NX1]\n","y_pred = np.dot(X_train, W) + b\n","print (f\"y_pred: {y_pred.shape}\")"],"execution_count":108,"outputs":[{"output_type":"stream","text":["y_pred: (35, 1)\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"hcvVc7UkWyMx","colab_type":"text"},"source":["## Loss"]},{"cell_type":"markdown","metadata":{"id":"8EbP8eWUoRqo","colab_type":"text"},"source":["3. Compare the predictions $\\hat{y}$ with the actual target values $y$ using the objective (cost) function to determine the loss $J$. A common objective function for linear regression is mean squarred error (MSE). This function calculates the difference between the predicted and target values and squares it.\n","\n","  * $J(\\theta) = MSE = \\frac{1}{N} \\sum_{i-1}^{N} (y_i - \\hat{y}_i)^2 $\n","    * ${y}$ = ground truth | $\\in \\mathbb{R}^{NX1}$\n","    * $\\hat{y}$ = predictions | $\\in \\mathbb{R}^{NX1}$"]},{"cell_type":"code","metadata":{"id":"h30Y6v5FoPBL","colab_type":"code","outputId":"3b8a3bb4-098b-4faf-cb05-2f5a9ae8b685","executionInfo":{"status":"ok","timestamp":1583941061922,"user_tz":420,"elapsed":495,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["# Loss\n","N = len(y_train)\n","loss = (1/N) * np.sum((y_train - y_pred)**2)\n","print (f\"loss: {loss:.2f}\")"],"execution_count":109,"outputs":[{"output_type":"stream","text":["loss: 0.99\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"BbzZqCXLWzuJ","colab_type":"text"},"source":["## Gradients"]},{"cell_type":"markdown","metadata":{"id":"yBdMI-Xxoazo","colab_type":"text"},"source":["4. Calculate the gradient of loss $J(\\theta)$ w.r.t to the model weights.\n","  * $J(\\theta) = \\frac{1}{N} \\sum_i (y_i - \\hat{y}_i)^2  = \\frac{1}{N}\\sum_i (y_i - X_iW)^2 $\n","    * $\\frac{\\partial{J}}{\\partial{W}} = -\\frac{2}{N} \\sum_i (y_i - X_iW) X_i = -\\frac{2}{N} \\sum_i (y_i - \\hat{y}_i) X_i$\n","    * $\\frac{\\partial{J}}{\\partial{W}} = -\\frac{2}{N} \\sum_i (y_i - X_iW)1 = -\\frac{2}{N} \\sum_i (y_i - \\hat{y}_i)1$"]},{"cell_type":"code","metadata":{"id":"U5zrZ2j4oPYX","colab_type":"code","colab":{}},"source":["# Backpropagation\n","dW = -(2/N) * np.sum((y_train - y_pred) * X_train)\n","db = -(2/N) * np.sum((y_train - y_pred) * 1)"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"_sIznBOQrFtw","colab_type":"text"},"source":["**NOTE**: The gradient is the derivative, or the rate of change of a function. It's a vector that points in the direction of greatest increase of a function. For example the gradient of our loss function ($J$) with respect to our weights ($W$) will tell us how to change W so we can maximize $J$. However, we want to minimize our loss so we subtract the gradient from $W$. "]},{"cell_type":"markdown","metadata":{"id":"zKyRWE7BXKku","colab_type":"text"},"source":["## Update weights"]},{"cell_type":"markdown","metadata":{"id":"4Hs2ikYNpSSd","colab_type":"text"},"source":["5. Update the weights $W$ using a small learning rate $\\alpha$.\n","  * $W = W - \\alpha\\frac{\\partial{J}}{\\partial{W}}$\n","  * $b = b - \\alpha\\frac{\\partial{J}}{\\partial{b}}$"]},{"cell_type":"code","metadata":{"id":"rKuOCcjep3ND","colab_type":"code","colab":{}},"source":["LEARNING_RATE = 1e-1"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"v9E7zB0IoPd0","colab_type":"code","colab":{}},"source":["# Update weights\n","W += -LEARNING_RATE * dW\n","b += -LEARNING_RATE * db"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"hYijMGqvpigf","colab_type":"text"},"source":["**NOTE**: The learning rate $\\alpha$ is a way to control how much we update the weights by. If we choose a small learning rate, it may take a long time for our model to train. However, if we choose a large learning rate, we may overshoot and our training will never converge. The specific learning rate depends on our data and the type of models we use but it's typically good to explore in the range of $[1e^{-8}, 1e^{-1}]$. We'll explore learning rate update stratagies in later lessons."]},{"cell_type":"markdown","metadata":{"id":"rRiMWUARXBM4","colab_type":"text"},"source":["## Training"]},{"cell_type":"markdown","metadata":{"id":"745G2w_rrBZl","colab_type":"text"},"source":["6. Repeat steps 2 - 5 to minimize the loss and train the model."]},{"cell_type":"code","metadata":{"id":"-QpmeHBhrIQI","colab_type":"code","colab":{}},"source":["NUM_EPOCHS = 50"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"89vkuOZdoPcK","colab_type":"code","outputId":"ef9e2465-ee8a-4a88-a1fd-5ca9101957bc","executionInfo":{"status":"ok","timestamp":1583941076742,"user_tz":420,"elapsed":528,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":102}},"source":["# Initialize random weights\n","W = 0.01 * np.random.randn(INPUT_DIM, OUTPUT_DIM)\n","b = np.zeros((1, ))\n","\n","# Training loop\n","for epoch_num in range(NUM_EPOCHS):\n","\n","    # Forward pass [NX1] · [1X1] = [NX1]\n","    y_pred = np.dot(X_train, W) + b\n","\n","    # Loss\n","    loss = (1/len(y_train)) * np.sum((y_train - y_pred)**2)\n","\n","    # show progress\n","    if epoch_num%10 == 0:\n","        print (f\"Epoch: {epoch_num}, loss: {loss:.3f}\")\n","\n","    # Backpropagation\n","    dW = -(2/N) * np.sum((y_train - y_pred) * X_train)\n","    db = -(2/N) * np.sum((y_train - y_pred) * 1)\n","\n","    # Update weights\n","    W += -LEARNING_RATE * dW\n","    b += -LEARNING_RATE * db"],"execution_count":114,"outputs":[{"output_type":"stream","text":["Epoch: 0, loss: 0.990\n","Epoch: 10, loss: 0.039\n","Epoch: 20, loss: 0.028\n","Epoch: 30, loss: 0.028\n","Epoch: 40, loss: 0.028\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"emgsY5TJXZ7R","colab_type":"text"},"source":["## Evaluation"]},{"cell_type":"code","metadata":{"id":"cNV_o2_arQNU","colab_type":"code","colab":{}},"source":["# Predictions \n","pred_train = W*X_train + b\n","pred_test = W*X_test + b"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"KMj2xv1xrXb5","colab_type":"code","outputId":"e2a06e43-7fa2-4115-eeec-dc09975479aa","executionInfo":{"status":"ok","timestamp":1583941079574,"user_tz":420,"elapsed":766,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["# Train and test MSE\n","train_mse = np.mean((y_train - pred_train) ** 2)\n","test_mse = np.mean((y_test - pred_test) ** 2)\n","print (f\"train_MSE: {train_mse:.2f}, test_MSE: {test_mse:.2f}\")"],"execution_count":116,"outputs":[{"output_type":"stream","text":["train_MSE: 0.03, test_MSE: 0.01\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"59OcpgFArYw_","colab_type":"code","outputId":"67b03206-5ac5-48d2-d925-14ac8d1dd9f1","executionInfo":{"status":"ok","timestamp":1583941081212,"user_tz":420,"elapsed":1758,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":336}},"source":["# Figure size\n","plt.figure(figsize=(15,5))\n","\n","# Plot train data\n","plt.subplot(1, 2, 1)\n","plt.title(\"Train\")\n","plt.scatter(X_train, y_train, label='y_train')\n","plt.plot(X_train, pred_train, color='red', linewidth=1, linestyle='-', label='model')\n","plt.legend(loc='lower right')\n","\n","# Plot test data\n","plt.subplot(1, 2, 2)\n","plt.title(\"Test\")\n","plt.scatter(X_test, y_test, label='y_test')\n","plt.plot(X_test, pred_test, color='red', linewidth=1, linestyle='-', label='model')\n","plt.legend(loc='lower right')\n","\n","# Show plots\n","plt.show()"],"execution_count":117,"outputs":[{"output_type":"display_data","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAA3IAAAE/CAYAAAADjvF6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOzdeXzU9bX/8fchRohrEHFJkEKtoigI\nGlfU1o1oayXQ2qu1FReK9ld7r124grbq7bWCRWttta3UWuy1rVaFSMUai1SxrqBRUBGlFiRB2RSL\nGiQk5/fHTDIzyUwyySzfWV7Px4MH3893vjPfMwnk5MxnM3cXAAAAACB/9Ak6AAAAAABAz1DIAQAA\nAECeoZADAAAAgDxDIQcAAAAAeYZCDgAAAADyDIUcAAAAAOQZCjkgx5lZiZl9aGaDg44FAAAAuYFC\nDkizcNHV9qfVzJqi2uf19PXcvcXdd3H3tzMRLwAA2ZbuXBn1us+a2dfSGSuQq3YIOgCg0Lj7Lm3H\nZrZK0iR3X5DoejPbwd23ZyM2AAByQU9zJYDO6JEDsszMrjOze83sT2a2RdLXzOzY8KeIm83sHTP7\nuZmVhq/fwczczIaE23eHH/+rmW0xs2fMbGiAbwkAgLQKTyv4oZm9ZWYbzewPZlYefmxnM7vHzN4L\n583nzKy/md0k6UhJd4R79m4K9l0AmUUhBwRjvKQ/Stpd0r2Stkv6L0l7Shoj6XRJl3Tx/K9K+qGk\nPSS9Lel/MxksAABZ9n1JYyUdL2mQpGZJN4cfm6TQqLJKhfLmZZK2ufv3JC1WqHdvl3AbKFgUckAw\n/uHuf3H3VndvcvfF7v6cu29397ckzZL02S6ef7+7L3H3Zkl/kDQqK1EDAJAdl0qa6u5r3X2rpP+R\n9B9mZgoVdQMl7R/Om4vd/aMggwWCwBw5IBhrohtmdpCkmyQdIWknhf5vPtfF89+NOv5Y0i6JLgQA\nIJ+Ei7X9JD1sZh71UB9JAyT9VtI+ku43s10k/V7SD929JevBAgGiRw4Ihndo3y7pFUmfcffdJF0t\nybIeFQAAAXN3l9Qo6WR3L4/608/dN7r7J+5+tbsfJOlESWdLOqft6UHFDWQbhRyQG3aV9IGkj8zs\nYHU9Pw4AgEL3a0kzzGw/STKzvczsi+HjU81suJn1kfRvheaZt4aft07Sp4MIGMg2CjkgN3xP0kRJ\nWxTqnbs32HAAAAjUTyQtkLQwvMLz05IODz9WKelBhXLmK5IeViRv3izpfDN738x+kt2QgeyyUO81\nAAAAACBf0CMHAAAAAHmGQg4AAAAA8gyFHAAAAADkGQo5AAAAAMgzFHIAAAAAkGd2CDqAruy5554+\nZMiQoMMAAGTYCy+8sNHdBwYdR74gPwJA8UiUI3O6kBsyZIiWLFkSdBgAgAwzs9VBx5BPyI8AUDwS\n5UiGVgIAAABAnqGQAwAAAIA8QyEHAAAAAHmGQg4AAAAA8gyFHAAAAADkGQo5AAAAAMgzaSnkzOxO\nM1tvZq8kePxzZvaBmb0U/nN1Ou4LAAAAAMUoXfvIzZZ0q6Tfd3HNk+5+ZpruBwAAAABFKy2FnLsv\nMrMh6XgtAEDuq61v1My6FVq7uUkV5WWaUj1MNaMrgw4LAICikc05csea2ctm9lczOySL9wUApFFt\nfaOmzVmmxs1NckmNm5s0bc4y1dY3Bh0aAADB27ZNamjI+G2yVci9KOlT7n6YpF9Iqk10oZlNNrMl\nZrZkw4YNWQoPAJCsmXUr1NTcEnOuqblFM+tWBBQRAAA54vXXpWOPlX7604zfKiuFnLv/290/DB8/\nLKnUzPZMcO0sd69y96qBAwdmIzwAQA+s3dzUo/MAABQ8d+mXv9Qnxx6nmYNP1NDSkzVmxsKMjlbJ\nSiFnZvuYmYWPjwrfd1M27g0ASK+K8jKNe/XvWnXDmTrpn4tjzgMAUHTefVf6whf0/m2zVHPuT3Tb\nsFPlZhmfepCWxU7M7E+SPidpTzNrkHSNpFJJcvdfS/qypG+a2XZJTZLOcXdPx70BAFn08cd68gdj\n1aelRUsqD9bjnz5CklRWWqIp1cMCDg4AgCyrrZUuvVT6xjc0bszlentLc8zDbVMPMrEgWLpWrTy3\nm8dvVWh7AgBAvpo9W7rwQvWR9MTsB3XlO7tIm5tUyaqVAIBi8+GH0ne+Iz32mDRnjnTccVozdX7c\nSzM19SBd+8gBAPJQUtsIfPihtOuuoeOTT5YWLNBnzfRU9sMFACB4zz4rff3r0gknSC+9JO22m6TQ\nFIPGOEVbpqYeZHP7AQBADklqG4GLL44UcYsXhz55DE15BgCguDQ3S9deK9XUSDfcIN15Z3sRJ0lT\nqoeprLQk5imZnHpAjxwAFKmuthGoGejSfvuFTg4YIG3YQAEHACheb74pfe1rUnm59OKLUkVFp0va\nRrR0O9IlTSjkAKBIJRqz/7sbL5SmvR1qzJ4tTZyYvaAAAMgl7tIdd0jTpknXXCN961tSn8SDGmtG\nV2ZtzjiFHAAUqY5j+Qd9sE7/+PXFkQtaW+mFAwAUr/XrpW98Q3r7bWnRImn48KAjisEcOQAoUtFj\n+Z/65YXtRdzi6b+U3FX70lqNmbFQQ6fOz/impgAA5JT586VRo6SDD5aeey7nijiJHjkAKFo1oytV\n/kq9Pnf+F9vP1b7YoJrRle0LobTNoWtbCKXteQAAFKSPP5a+//1QIfenP0mf/WzQESVEjxwAFCuz\nSBF3662Se8xE7UQLoQAAUJCWLJFGj5a2bJGWLs3pIk6iRw4Ais+iRbHJyb3TJYkWQsnUpqYAACQr\nqT1Qe6KpKbRP6j//Kf3iF9J//Ef6gs0geuQAoJiYRYq4n/88bhEnJd68NFObmgIAkIyk9kDtiTvu\nkHbaKbTJ94sv5k0RJ1HIAUBxePTR2BUo3aVvfzvh5dne1BQAgGSkbei/eygvfuMbofa2bdKgQWmK\nMjso5ACgANTWNyZeYdJMqq4OHf/2twl74aLVjK7U9AkjVFleJpNUWV6m6RNGsNAJACBQaRn6/8wz\nkb3gpk4N5cXS0jREl13MkQOAPJdohckh9/1eo6ZfGbkwiQIuWjY3NQUAIBkd90CNPp+UkSOlZaFV\nmLVunbTXXmmMLrvokQOAPJGo1y3eMJPl150RKeKmT+9xEQcAQC7q9dD/tWtDI1SWLZOOPjqUF/O4\niJPokQOAvNDVvm7Rw0kmP/eArnz8d5EnUsABAApI9DY5Sa9aefnl0i23hI6XLJGOOCILkWYehRwA\n5JBESyp3Nbm7bZjJqhvObH/sh6ddqoUnn62nsv0G0ImZnS7pFkklku5w9xkdHr9A0kxJbRMbb3X3\nO7IaJADkkaSH/n/yidSvX6Td2hq78Feeo5ADgByRbK9btLWbm/TEK7/T4PkPtJ8bcsVDKist0XRW\nmAycmZVIuk3SaZIaJC02s3nu/lqHS+9198uyHiAAFKo//EH62tdCx7//vfT1rwcbTwZQyAFAjkim\n162jf0X1wv2qepJ+MqpGlenYHBXpcpSkle7+liSZ2T2SxknqWMgBANLBPbIipSRt3Sr17RtcPBlE\nIQcAOaKrXreb/2NUTG/dbx74kU5b+XzkInd9U9I3sxAneqRS0pqodoOko+Nc9yUzO1HSG5K+4+5r\n4lwDAOjKiy9G5r/9539G5sUVKFatBIAckWjp5Irysph93VbdcGakiLv1VhY0yX9/kTTE3UdK+puk\nu+JdZGaTzWyJmS3ZsGFDVgMEgJw3ZkykiGtsLPgiTqKQA4Cc0d2SyjVjR+mpaadEHnSXvvWtbIaI\nnmuUtF9Ue5Aii5pIktx9k7t/Em7eISnucmruPsvdq9y9auDAgRkJFgDyzoYNoQVMnn5aOuSQUG6s\nqAg6qqygkAOAHBHd62aSKsvLNH3CiNBcNzNp48bQhb/5Db1w+WOxpAPMbKiZ7SjpHEnzoi8ws32j\nmmdJWp7F+AAgf111VWQvuKeekl55Jdh4sow5cgCQQzotqdxxmWQKuLzi7tvN7DJJdQptP3Cnu79q\nZj+StMTd50n6TzM7S9J2Se9JuiCwgAEgH2zdKpVFTUcosG0FkkWPHADkIvfYpHTnnRRxecrdH3b3\nA919f3f/cfjc1eEiTu4+zd0PcffD3P0kd3892IgBIIddcUWkiLv66s75sodq6xs1ZsZCDZ06X2Nm\nLFRtfWP3T8oR9MgBQBcSbdCdUfTCAQDQWXR+fP99qbw8pZfrav/WfNjChx45AEig7Qd84+YmuSI/\n4DP2aV3HoSH33UcRBwDAX/8amx/dUy7ipK73b80HaemRM7M7JZ0pab27HxrncZN0i6TPS/pY0gXu\n/mI67g0AmdLVD/i0f1KXwV64QHoVAQBIh+j8uHixVFWVtpfuav/WfJCuHrnZkk7v4vEzJB0Q/jNZ\n0q/SdF8AyJis/IDfvj02Sd1/f9qLuKz2KgIAkA5r1nTuhUtjESd1vX9rPkhLIefuixRaaSuRcZJ+\n7yHPSirvsNwyAOScjP+AN5NKSyNtd+lLX0rPa4fl+7ARAEARqqqSBg8OHd96a8amGXS3f2uuy9Yc\nuUpJa6LaDeFzAJCzMvYDvqkp9lPGRx7JWJLK92EjAIAi0twcyo8vvBBqt7RI3/pWxm7X5f6teSDn\nVq00s8kKDb/U4LZKHAAC0PaDPK3zy7K8ImVFeZka4xRt+TJsBABQJK67TvrhD0PHY8dKdXVZuW2n\n/VvzSLYKuUZJ+0W1B4XPdeLusyTNkqSqqiqWawMQqLT9gP/3v6Xdd4+0FyyQTjkl9dftxpTqYTFL\nK0v5NWwEAFAEoj/kXL9eGjgwuFjySLYKuXmSLjOzeyQdLekDd38nS/cGgGAFuC9cRnoVAQBIhyee\nkD73uUibLXd6JF3bD/xJ0uck7WlmDZKukVQqSe7+a0kPK7T1wEqFth+4MB33BYBkBbIE/4YN0l57\nRdrPPScddVRm7xlHPg8bAQAUqOgPOZ98Ujr++OBiyVNpKeTc/dxuHndJmZupCABdaFuCv214YdsS\n/JIyV+AE2AsHAEDOWrdO2mefSJv82GvZWrUSAAKT1SX4//Wv2CJu6VKSFAAAUmhueFsRN306+TFF\nObdqJQCkW9aW4KcXDgCAzlpapB2iyo7m5tg2eoUeOQAFL+Mbe7/6amwR98YbFHEAAEjSz34WKdqO\nPjqUHyni0oKvIoCCl9El+OmFAwAgvugc2dAgVbLwVjrRIweg4NWMrtT0CSNUWV4mk1RZXqbpE0ak\nttBJfX1sglqzhiIOAABJev752BzpThGXAfTIASgKaV2Cn144AADii86RdXXS2LHBxVLg6JEDgGQ9\n+WRsgnrvPYo4AACkUE7s2AtHEZdRFHIAkAwz6cQTI213qX//4OIBACBXfOlL0oABoeMrr+RDzixh\naCUAdOX++6Wzz460P/xQ2nnn4OIBACBXuEt9ovqFtm6V+vYNLp4iQyEHAImkMBeutr5RM+tWaO3m\nJlWUl2lK9bD0zdEDACBod9whfeMboeMDD5RWrAg2niJEIQcAHUUnJ6nHnzDW1jfGbHfQuLlJ0+Ys\nkxRadIUiDwCQ16I/6HzrLWno0OBiKWIUcgAQLQ0rUs6sWxGzZ50kNTW3aGZd6NPKroo8AABy1tKl\n0mGHRdrMhQsUi50AgCT95CexRVxzc68T1NrNTQnPd1fkAQCQk/r2jRRxc+dSxOUAeuQAIM37wlWU\nl6kxTjFXUV7WZZEHAEDO2bJF2m23SJsCLmfQIwegeF1xRWwR19qalgQ1pXqYykpLYs6VlZZoSvUw\nVZSXxX1OovMAAATmoosiRdy3v00Rl2PokQNQnNLcCxetba5bogVNoufISZEiDwCAnNBxW4GPPpJ2\n2im4eBAXhRyAvJLyio8XXijNnh1pZ+jTxZrRlXHj6q7IAwAgUH/6k/TVr4aOBw6U1q8PNh4kRCEH\nIG90t6x/tzLYC9cTiYo8AAACFZ0nly+XDjoouFjQLebIAcgbvV7x8cwzY5OTO+P8AQBo88YbnfMk\nRVzOo0cOQN7o1YqPGeyFY2NvAEDe23df6d13Q8d33y2dd16w8SBpFHIA8kZXy/p3UlUlvfBCpJ3m\nHriUh3kCABCkjz+Wdt450m5t7fzhJ3IaQysB5I2ulvWPYZbRIk5KYZgnAABBu/zySBF3/vmhPEkR\nl3fokQOQN7pd8bGiQnrnncgTMjgPjo29AQB5Kbpg++CD2M2+kVco5ADklYQrPmZ5RcoeDfMEACBo\n8+ZJ48aFjktKpO3bg40HKWNoJYD81qdPICtSJj3MEwCAoJlFirj6eoq4AkGPHID8FV3A9e0rbd2a\ntVuzsTcAIOetXi0NGRJps/VOQUlLIWdmp0u6RVKJpDvcfUaHxy+QNFNSY/jUre5+RzruDaAIsbE3\nAABdGz48tKm3JN1+uzR5crDxIO1SLuTMrETSbZJOk9QgabGZzXP31zpceq+7X5bq/QDkh4ztsRZd\nxO2/v7RyZeqvCQBAodi2LTRKpU1LS2gaAgpOOr6rR0la6e5vufs2SfdIGpeG1wWQp9r2WGvc3CRX\nZI+12vrGbp+bkFnnuXAUcQAARFx9daSIGzculCsp4gpWOr6zlZLWRLUbwuc6+pKZLTWz+81svzTc\nF0COSvsea9EF3JgxjPEHAKAjM+l//zd0vHGjVFsbbDzIuGyV6H+RNMTdR0r6m6S7El1oZpPNbImZ\nLdmwYUOWwgOQTmnbYy1eL9w//pFCZAAAFJgFCzrnygEDgosHWZOOQq5RUnQP2yBFFjWRJLn7Jnf/\nJNy8Q9IRiV7M3We5e5W7Vw0cODAN4QHItkR7qfVoj7XopDRhAr1wAAB0ZCaddlro+JlnyJVFJh2F\n3GJJB5jZUDPbUdI5kuZFX2Bm+0Y1z5K0PA33BZCjUtpjLV4v3AMPpDlCIHvM7HQzW2FmK81sapzH\n+5rZveHHnzOzIdmPEkBeWbu2c6485pjg4kEgUi7k3H27pMsk1SlUoP3Z3V81sx+Z2Vnhy/7TzF41\ns5cl/aekC1K9L4DcVTO6UtMnjFBleZlMUmV5maZPGNH1qpXusUnp0kv5ZBF5L2pl5zMkDZd0rpkN\n73DZxZLed/fPSLpZ0g3ZjRJAXhkzRqoM59ObbiJXFrG07CPn7g9LerjDuaujjqdJmpaOewHIrt5u\nI9CjPdZyZF84IAPaV3aWJDNrW9k5eouecZKuDR/fL+lWMzN3/iMAiLJ9u1RaGtsuKUl8PQoe65EC\nSCgj2whEa22NLeJ+8AOKOBSaZFZ2br8mPMrlA0msVAAgYubMSBH32c+GciVFXNFLS48cgMLU1TYC\nKW/uTS8c0CNmNlnSZEkaPHhwwNEAyJrofPnOO9I++wQXC3IKPXIAEkrbNgLRmptjk9LNN1PEoZB1\nu7Jz9DVmtoOk3SVt6vhCrOoMFJmnnuq8oAlFHKLQIwcgoYryMjXGKdp6tI1ANHrhUHzaV3ZWqGA7\nR9JXO1wzT9JESc9I+rKkhcyPA4pcdL5cuFA66aTgYkHOokcOQEIpbSMQrakpNinNnk0Rh6KQ5MrO\nv5U0wMxWSvqupE5bFAAoEhs3du6Fo4hDAvTIAUiobR5cb1atbEcvHIpcEis7b5V0drbjApBjvvAF\n6eHwj4prr5WuuSbQcJD7KOQAdKlH2whE27JF2m23SLu2Vho3Ln2BAQBQCFpbY1eg3LYtdpsBIAGG\nVgJIP7PYIs6dIg4AgI5+9atIETdyZChfUsQhSfTIAUifjRul6NX0nnpKOu644OIBACBXRU89WL1a\nYlsR9BCFHID0YC4cAADde/FF6YgjIm3yJXqJoZUAUtPYGFvEvfwySQkAgHjMIkXcQw+RL5ESeuQA\n9F6HXrjaFxtUM7IXC6MAAFDIPvhAKi+PtCngkAb0yAHouTffjCniTpz8Gw254iFNm7NMtfWNAQYG\nAECOOe+8SBH3ve9RxCFt6JED0DMdeuGGXPFQ+3FTc4tm1q3o3XYFAAAUEnepT1SfSVOT1K9fcPGg\n4FDIAUjOyy9Lo0a1N4/+f3dp3a4DOl22dnNTUi9XW9+Y2kbjAADkqt//Xpo4MXS8337S228HGw8K\nEoUcgO7FWZFyhxkLpThFW0V5WbcvV1vfqGlzlqmpuUWS1Li5SdPmLJMkijkAQH6LzplvvCEdcEBw\nsaCgMUcOKCC19Y0aM2Ohhk6drzEzFiY9Xy3h8/7+99iEtHFj+9j+KdXDVFZaEvM6ZaUlmlI9rNv7\nzaxb0V7EtWkblgkAQF567bXYnOlOEYeMokcOKBC97eVK9LyawwfFXthhcnbba/ZmeGSi4ZfJDssE\nACCn9O8vbd4cOv7zn6Wzzw42HhQFCjmgQHTVy9VVcdXxeWPfeEaz5v44csEHH0i77Rb3uTWjK3s1\nFLKivEyNvRyWCQBAzvjoI2mXXSLt1tbO0xGADGFoJVAgetvLFf34qhvOjC3i3BMWcalIZVgmAAA5\n4ZvfjBRxkyaFciZFHLKIHjmgQPS2l6uivEzHPvkX3fjwz9rPDfvuA9pzYLmeSnuUIakMywQAIFAd\ntxXYsiW2Vw7IEgo5oEBMqR4WM9dNSq6X66lpp8S0h1zxUFZ6x3o7LBMAgMDMmSN96Uuh4112CRVx\nQEAo5IAC0eNerptvlr773fbmidc9qjVbtqmS3jEAADqLHja5bJl06KHBxQKIQg4oKEn3csXZF25R\nZkICACC//fOf0mc+E2l3WMUZCAqLnQAB6e2ebym56qrYIq6lhYQEAEAi++8fKeJ+9ztyJnIKPXJA\nAHq751tK4vTCAQCAOLZulcqiFgtjWwHkoLT0yJnZ6Wa2wsxWmtnUOI/3NbN7w48/Z2ZD0nFfIF91\ntedb2l16aWzyaW2liAMAIJErrogUcV/5CtsKIGel3CNnZiWSbpN0mqQGSYvNbJ67vxZ12cWS3nf3\nz5jZOZJukPQfqd4byFe93fOtx+iFAwAgedF58/33pfLy4GIBupGOHrmjJK1097fcfZukeySN63DN\nOEl3hY/vl3SKGR9toHgl2tutuz3fkjZhQmwycqeIAwAgkUce6Zw3KeKQ49JRyFVKWhPVbgifi3uN\nu2+X9IGkAWm4N5CXplQPU1lpScy5tO3dZibNnRtpU8ABAJCYmXTGGaHjxYvJm8gbObdqpZlNNrMl\nZrZkw4YNQYcDZETN6EpNnzBCleVlMkmV5WWaPmFEagudHHccvXAAACSroaFz3qyqCi4eoIfSsWpl\no6T9otqDwufiXdNgZjtI2l3Spngv5u6zJM2SpKqqKn4LRcFKes+3ZDAXDgCA5FVVSS+8EDr+xS+k\nyy4LNh6gF9JRyC2WdICZDVWoYDtH0lc7XDNP0kRJz0j6sqSF7vymCaRs6FBp1apIm/9WAAAk1tws\n7bhjpN3SIvXJuQFqQFJS/pcbnvN2maQ6Scsl/dndXzWzH5nZWeHLfitpgJmtlPRdSZ22KADQQ2YU\ncQAAJOvHP44UcWPHhvImRRzyWFo2BHf3hyU93OHc1VHHWyWdnY57AUWPYZQAAPRMdO5cv14aODC4\nWIA04WMIIJ9QxAEAkLwnnui8oAlFHApEWnrkAGQYBRwAAD0TnTsXLZJOOCG4WIAMoJADcl10IurT\nR7VL3tbMGQu1dnOTKsrLNKV6WPpWvwQAIN+tWyfts0+kzYefKFAMrQRylVmn4SC1S97WtDnL1Li5\nSS6pcXOTps1Zptr6jjt+AABQhE49NVLETZ9OEYeCRo8ckIuiC7jBg6XVqyVJM+tWqKm5JebSpuYW\nzaxbQa8cAKB4tbRIO0T9WtvcHNsGChA9ckAuidML11bESdLazU1xn5boPAAABe+WWyJF29FHh3In\nRRyKAP/KgVwRXcAdc4z0zDOdLqkoL1NjnKKtorwsk5EBAJCbonNnQ4NUyegUFA965ICgxeuFi1PE\nSdKU6mEqKy2JOVdWWqIp1cMyGSEAALnl+ec7506KOBQZCjkgKO6xSWj8+G4nZdeMrtT0CSNUWV4m\nk1RZXqbpE0YwPw4AUDzMQkMoJamujgVNULQYWgkEIYV94WpGV1K4AQCKz3vvSQMGRNoUcChy9MgB\n2dSxF27qVBIRAADd+fKXI0XclVeSOwHRIwdkTwq9cAAAFCV3qU9Uv8PWrVLfvsHFA+QQeuSATNu+\nPbaImzmTIg4AgO789reRIu7AA0O5kyIOaEePHJBJ9MIBANBz0fnzrbekoUODiwXIUfTIAZmwdWts\nEvrtbynigCJjZnuY2d/M7M3w3/0TXNdiZi+F/8zLdpxATlm6tPO2AhRxQFz0yKGo1NY3ambdCq3d\n3KSK8jJNqR6W/hUg6YUDEDJV0mPuPsPMpobbV8S5rsndR2U3NCAH9e0rbdsWOp47V6qpCTYeIMfR\nI4eiUVvfqGlzlqlxc5NcUuPmJk2bs0y19Y3pucGWLbFF3AMPUMQBxW2cpLvCx3dJ4rdSIJ62/NlW\nxLlTxAFJoJBD0ZhZt0JNzS0x55qaWzSzbkW3z62tb9SYGQs1dOp8jZmxsHPxZybttluk7S5NmJCO\nsAHkr73d/Z3w8buS9k5wXT8zW2Jmz5oZv72iuFx8cSR/XnYZH4ACPcDQShSNtZubenS+TVtPXlsR\n2NaTJ0k1g3aU9torcvGCBdIpp6QnYAA5z8wWSNonzkNXRTfc3c0s0W+on3L3RjP7tKSFZrbM3f8Z\n516TJU2WpMGDB6cYORCwjtsKfPSRtNNOwcUD5CEKORSNivIyNcYp2irKy7p8XqKevJrDB8VeyKeI\nQNFx91MTPWZm68xsX3d/x8z2lbQ+wWs0hv9+y8welzRaUqdCzt1nSZolSVVVVfzAQf665x7p3HND\nx3vuKW3YEGw8QJ5iaCWKxpTqYSorLYk5V1ZaoinVw7p8Xsceu4p/r9eqG86MnHj6aYo4APHMkzQx\nfDxR0oMdLzCz/mbWN3y8p6Qxkl7LWoRAtplFirjlyynigBTQI4ei0bY6ZU9XrYzuyYsp4CQKOABd\nmSHpz2Z2saTVkr4iSWZWJelSd58k6WBJt5tZq0Ifrs5wdwo5FJ433pCGRX1wSv4EUkYhh6JSM7qy\nx9sNTKkepl/dUae6X17cfu6syb/URZeexRJ0ABJy902SOk2adfclkiaFj5+WNCLLoQHZte++0rvv\nho7vvls677xg4wEKBIUc0NQp0IMAACAASURBVI2awwfFFGxjpj+Wmf3nAAAoJE1NsQuYtLZ23msV\nQK8xRw5IZOnS2ISzerXkrqemnkwRBwBAV77znUgR97WvhYZSUsQBaUWPHBBPx2TDWH4AAJITnUM/\n+CB2n1UAaZNSj5yZ7WFmfzOzN8N/909wXYuZvRT+My+VewIZ9fTTsQlo3TqKOAAAkvGXv0RyaJ8+\nofxJEQdkTKo9clMlPebuM8xsarh9RZzrmtx9VIr3AjKLXjgAAHonOofW10uj+LUPyLRU58iNk3RX\n+PguiUX8kIceeSQ2AW3eTBEHAEAyVq+OzaHuFHFAlqRayO3t7u+Ej9+VtHeC6/qZ2RIze9bMKPaQ\nO8ykM86ItN2l3XcPLh4AAPLFIYdIQ4aEjn/9az4EBbKs26GVZrZA0j5xHroquuHubmaJ/gd/yt0b\nzezTkhaa2TJ3/2eC+02WNFmSBg8e3F14QO/ce690zjmR9scfS2VlwcUDAEC+2LZN6ts30m5pCc2J\nA5BV3f6vc/dT3f3QOH8elLTOzPaVpPDf6xO8RmP477ckPS5pdBf3m+XuVe5eNXDgwF68JaAbZrFF\nnDtFHAAAybj66kgRd9ZZoRxKEQcEItX/efMkTQwfT5T0YMcLzKy/mfUNH+8paYyk11K8L9Bzs2bF\njuPfto1hIAAAJMtM+t//DR1v3Cg92OnXPgBZlOqqlTMk/dnMLpa0WtJXJMnMqiRd6u6TJB0s6XYz\na1WocJzh7hRyyC5WpAQAoHcee0w69dRImxwK5ISUCjl33yTplDjnl0iaFD5+WtKIVO4D9NqMGdK0\naZF2Bsbx19Y3ambdCq3d3KSK8jJNqR6mmtGVab0HAACBiP4g9JlnpGOOCS4WADFS7ZEDclcWeuFq\n6xs1bc4yNTW3SJIaNzdp2pxlkkQxBwDIX2vXSpVReYxeOCDnMDsVhef7348t4lpbM5aAZtataC/i\n2jQ1t2hm3YqM3A8AgIw7/vhIEXfjjRRxQI6iRw6FJctz4dZuburReQAActb27VJpaWy7pCS4eAB0\niR45FIYLL4wt4tyz8gliRXn8bQsSnQcAICfNnBkp4k48MZRDKeKAnEaPHPJfgCtSTqkeFjNHTpLK\nSks0pXpY1mIAACAl0Xl07Vpp332DiwVA0uiRQ/4aPz6QXrhoNaMrNX3CCFWWl8kkVZaXafqEESx0\nAgDIfU8/3TmPUsQBeYMeOeSnHNoXrmZ0JYUbACC/ROfRxx6TTj45uFgA9Ao9csgvX/hC4L1wAADk\nrY0bO+dRijggL1HIIX+YSQ8/HDrebTcKOAAAeuLMM6WBA0PH11xDHgXyHEMrkfuOPlp6/vlIm8QD\nAEDyWltjV6Dcti12mwEAeYkeOeSs2vrGUC9cuIj7cPBQijgAABKorW/UmBkLNXTqfI2ZsTCUR3/1\nq0gRN2JEKI9SxAEFgR455KSNo49WzUuRXrghVzykstISTa9vZGERAAA6qK1vjNkOp3Fzk2oOHxS5\nYPVqafDggKIDkAn0yCH3mGnPcBH30EEnaMgVD0mSmppbNLNuRZCRAQCQk2bWrWgv4g55d6VW3XBm\n5EF3ijigANEjh9xx0EHSikih1lbARVu7uSmbEQEAkBfa8mN0AXfhl6/R4/sfqX8FFRSAjKKQQ26I\nXgr5m9/UmMFfluIUbRXlZVkMCgCA/HBAvxY9+j/j2tttH4ZWkjeBgsXQSgSrf//O+9n88peaUj1M\nZaUlMZeWlZZoSvWwLAcIAECOO++89iLuN0fWtBdx5E2gsNEjh7SorW/UzLoVWru5SRXlZZpSPaz7\nRUmiC7grr5R+/OP2Zttze/yaAAAUC3epT+Qz+XnPrNTsx1fLyJtAUaCQQ8rirZQ1bc4ySYqfQKIL\nOCnhlgI1oytJQAAAxPN//yedf37oeNAgac0anSXprGP2DzQsANnD0EqkLHqlrDYJV5iMLuJmzmRf\nOAAAesosUsS98Ya0Zk2w8QAIBD1ySFmilSRjzifZCwcAABJ47TXpkEMibXIpUNTokUPKEq0kWVFe\nFkoy0UXcrFkkHgAAemqPPSJF3L33kksB0COH1E2pHhYzR04KrZT11LRTpGlRF5J0AADomY8+knbZ\nJdJube08ygVAUaJHDkmrrW/UmBkLNXTqfI2ZsVC19Y2SQouSTJ8wQpXlZTJJg3brq+XXnRF54p/+\nRBEHAEBP/b//FyniJk3qPMoFQFGjRw5J6W5lyvYVJpkLBwBA6qLz6ZYtsb1yACB65JCkblembG6O\nTTrz51PEAQDQU3PmRPLpTjuFcilFHIA46JFDUrpcmTKAXrhebUAOAEAui86ny5ZJhx4aXCwAcl5K\nPXJmdraZvWpmrWZW1cV1p5vZCjNbaWZTU7knghFvZcq+27fpXzecGTnxxBNZK+KmzVmmxs1NckWG\nebbN2QMAIK/885+xRZw7RRyAbqU6tPIVSRMkLUp0gZmVSLpN0hmShks618yGp3hfZNmU6mEqKy1p\nb6+64UytuGlC5AJ36cQTsxJLjzYgBwAgl+2/v/SZz4SO77yTaQkAkpbS0Ep3Xy5J1vUKSkdJWunu\nb4WvvUfSOEmvpXJvdC3dQw/bnnvbvHr97dovRh5YvFiqStgZmxFJbUAOAEAu27pVKosa7cK2AgB6\nKBtz5ColrYlqN0g6Ogv3LXiJirXuVpjsrZrDB6km+kRAnxpWlJepMU7RlmhjcgAAcsrUqdINN4SO\nzz5b+vOfg40HQF7qtpAzswWS9onz0FXu/mC6AzKzyZImS9LgwYPT/fIFo6tirauhh70q5DZvlvr3\nj7RXrgwNBQlIog3Ip1QPCywmAACSEt3r9v77Unl5cLEAyGvdzpFz91Pd/dA4f5It4hol7RfVHhQ+\nl+h+s9y9yt2rBg4cmOQtik9XxVpahx6axRZx7oEWcVLnDcgry8s0fcIIVq0EkFNYEAwxHnmk84Im\nFHEAUpCNoZWLJR1gZkMVKuDOkfTVLNy3oHVVrKVl6OH69dLee0faa9ZIgwb1NMyMad+AHAByV9uC\nYLcnuiBqQbDTFJp6sNjM5rk788gLSXQB9/zz0pFHBhcLgIKR6vYD482sQdKxkuabWV34fIWZPSxJ\n7r5d0mWS6iQtl/Rnd381tbCRqChrmysXvcKk1MOhh2axRZx7ThVxAJAP3H25u3e3nG77gmDuvk1S\n24JgKAQNDZ174SjiAKRJSoWcu89190Hu3tfd93b36vD5te7++ajrHnb3A919f3f/capBo/N2AFKk\nWOv10MM1a2ISzheuekC1LzZkIHoAQFi8BcEYblAIqqqk/cIzS37+c7YVAJB22RhaiQxoK8oSbTHQ\n46GHHZY8HnLFQ9J2pWW1SwAoVNlcEIzFwPJEc7O0446RdkuL1CfVbXsBoDMKuTyWlnliK1dKBxzQ\n3hxx+b3a0nfn9nZKq10CQIFz91NTfImkFwRz91mSZklSVVUV3Tu56Mc/ln7wg9Dx2LFSXV2w8QAo\naBRyxaxDL9zQKx5SvN8M2GgbADKGBcEKRXROXbdO2muv4GIBUBTo6y9Gy5bFJpwPP5Tcu1xABQDQ\nMywIViSeeKLzgiYUcQCygB65YtOhFy568jUbbQNA+rj7XElz45xfKylmQTBJD2cxNKRLdE5dtEg6\n4YTgYgFQdCjkisXzz0tHHx1pb90q9e0bc0l3C6gAAAB13muVFSkBBIBCrhh00QvXERttAwDQhdNO\nkxYsCB1ff700bVqw8QAoWhRyhezxx6WTToq0m5ulHfiWAwDQYy0tsTmUnAogYCx2UqjMYos4dxIO\nAAC98fOfR3LoUUeRUwHkBH4KFZpFi6TPfjbSZiNSAAB6L3p6wpo10qBBwcUCAFH4Db+QmEWKuEMO\nCX1iSBEHAEDPPf98520FKOIA5BB+yy8EjzwSm2xaW6VXXgkuHgAA8plZZKXnujpWpQSQkxhame+i\nC7jjj5eefDK4WAAAyGfvvy/tsUekTQEHIIfRI5evnnii85APijgAAHrnv/87UsRNm0YRByDn0SOX\nj6ILuO99T7rxxuBiAQAgn3WcT/7JJ9KOOwYXDwAkiR65fNJxLpw7RRwAAL314IORIm78+FBepYgD\nkCfokcsX0QXctddK11wTWCgAAOS96Lza0CBVVgYXCwD0AoVcrpszR/rSlyJtxuwDAApcbX2jZtat\n0NrNTaooL9OU6mGqGZ2mQuvNN6UDD4y0yasA8hRDK3OZWaSIu/FGkg0AoODV1jdq2pxlatzcJJfU\nuLlJ0+YsU219Y+ovfuyxkSLu738nrwLIa/TI5aK775a+/vVIm0QDACgSM+tWqKm5JeZcU3OLZtat\n6H2v3McfSzvvHGmTVwEUAHrkco1ZpIj79a9JNgCAorJ2c1OPznfrRz+KFHEzZpBXARQMeuRyxaxZ\n0iWXRNokGgBAEaooL1NjnKKtorysZy/UcVuBpiapX78UowOA3EGPXC4wixRxv/89RRwAoGhNqR6m\nstKSmHNlpSWaUj0s+Rd59NFIEXfyyaG8ShEHoMDQIxekuXOlCRMibQo4AECRa5sH1+tVK6O3FXjr\nLWno0KSeltGVMgEgAyjkghKdaJ58Ujr++OBiAQAgh9SMrux5EbV6tTRkSKTdgw9H21bKbFtkpW2l\nzLZYACAXMbQy2/74x9gizp0iDgCAVJx+eqSIe/jhHo9w6WqlTADIVSn1yJnZ2ZKulXSwpKPcfUmC\n61ZJ2iKpRdJ2d69K5b55K7qAW7xYqirOLwMAAGnxySexc99aW2NzbZLSvlImAGRBqj1yr0iaIGlR\nEtee5O6jirKIu/POzr1wFHEAAPTeT38aKeJ+8INQbu1FESclXhGzxytlAkAWpdQj5+7LJcl6+YOz\nKER/bV5+WRo5MrhYAAAoBNG5dcsWaZddUnq5KdXDYubISb1YKRMAsixbc+Rc0qNm9oKZTc7SPYN1\n222de+Eo4gAA6L0nn4zk1iOOCOXWFIs4KbSgyfQJI1RZXiaTVFlepukTRrDQCYCc1m2PnJktkLRP\nnIeucvcHk7zP8e7eaGZ7Sfqbmb3u7nGHY4YLvcmSNHjw4CRfPod03ID0tdekgw8OLh4AAApBSUlo\nDpwkvf66NCy9vWW9WikTAALUbSHn7qemehN3bwz/vd7M5ko6Sgnm1bn7LEmzJKmqqiq/NlabOVP6\n7/+OtNkXDgCA1LzzjlRREWmTWwFAUhaGVprZzma2a9uxpLEKLZJSONomWLcVcStX9nj/mjEzFmro\n1PkaM2OhausbMxQoAAB55CtfiRRxDzxAEQcAUVLdfmC8pF9IGihpvpm95O7VZlYh6Q53/7ykvSXN\nDS+IsoOkP7r7IynGnTt+9CPpmmtCx2aRYR9JYhNSAAA6aGmRdtghtt2HrW8BIFqqq1bOlTQ3zvm1\nkj4fPn5L0mGp3CcntbaGxuu3Wb1a6sWcvq42IaWQAwAUnSVLpCOPDB1/97vSTTcFGw8A5Cg+3uqN\nqVMjRdzuu4eGevRyYRY2IQUAIOyrXw0VcSUlUlMTRRwAdCGlHrmi03Gox9q10r77pvSSFeVlaoxT\ntLEJKQCgaDQ2SoMGhY5//nPp298ONh4AyAP0yCXrsssiRdzgwaFeuBSLOCm0CWlZaUnMOTYhBQAU\njRtuiBRx775LEQcASaJHrjvNzdKOO0ba69dLAwem7eXb5sHNrFuhtZubVFFepinVw5gfBwAobB9/\nLO28c+h40iTpN78JNh4AyDMUcl258EJp9uzQ8fDh0quvZuQ2bEIKACgq8+ZJ48aFjl9+WRo5Mth4\nACAPUcjFs327VFoaab/3ntS/f3DxAABQCFpbpVGjpGXLpMMOk158kW0FAKCXiraQq61vjD+c8fbb\npUsvDV109NHSs88GGyiAvNXc3KyGhgZt3bo16FByRr9+/TRo0CCVRn9YhuLw8suhIk4K9ch98YvB\nxgMgZ5AvQ3qaI4uykIu3CffV972omsMHRS5qapL69QsoQgCFoKGhQbvuuquGDBkiMws6nMC5uzZt\n2qSGhgYNHTo06HCQTRdfLN15Z+j4o4+knXYKNh4AOYV82bscWZTjGTpuwj3xhb9o6fSzQo2f/jS0\nIiVFHIAUbd26VQMGDCjapNSRmWnAgAFF/4lrUVm3TjILFXEzZ4byK0UcgA7Il73LkUXZI9e22faO\n25v1xk3j288f+P25euM7NYmHXQJADxVzUoqHr0cRueUW6fLLQ8cNDVIleRRAYuSHnn8NirJHrqK8\nTJ/e1NBexP3gtG9qyBUPaeCA3dqHXTZubpIrNOxy2pxlqq1vDDZoAAjYkCFDtHHjxpSvQYHbujXU\nC3f55dJ554V64SjiABSJbObK4uuRa2nR7e8uVMUff6FfHzVBN554vraX7NC+CXfHYZeS1NTcopl1\nK+iVAwAUjIyMPnnkEemMM0LHS5ZIRxyReqAAgLiKq0futdek447Tocue0ZJ7H9H/jf+WWkp2UGV5\nmaZPGKGa0ZXtwy47SnQeAHLZqlWrdNBBB+mCCy7QgQceqPPOO08LFizQmDFjdMABB+j555/Xe++9\np5qaGo0cOVLHHHOMli5dKknatGmTxo4dq0MOOUSTJk2Su7e/7t13362jjjpKo0aN0iWXXKKWlpZE\nISAHpX30ibt0zDGhIu6AA6SWFoo4AHkjX3NlcRRy27dL06dLn/2sdNFF0oIFGnvmsXpq6sn614wv\n6KmpJ7d/CllRXhb3JRKdB4Bct3LlSn3ve9/T66+/rtdff11//OMf9Y9//EM33nijrr/+el1zzTUa\nPXq0li5dquuvv17nn3++JOl//ud/dPzxx+vVV1/V+PHj9fbbb0uSli9frnvvvVdPPfWUXnrpJZWU\nlOgPf/hDkG8xJ5nZ2Wb2qpm1mllVF9etMrNlZvaSmS3JRmxdjT7psddeC+0F99xz0n33SW+8wd5w\nAPJOPubKwh9a+cor0oUXhjb0XrJE+tSnurx8SvWwmK0JJLUPuwSAlGRiInfUJ3+JDB06VCNGjJAk\nHXLIITrllFNkZhoxYoRWrVql1atX64EHHpAknXzyydq0aZP+/e9/a9GiRZozZ44k6Qtf+IL69+8v\nSXrsscf0wgsv6Mgjj5QkNTU1aa+99kr/e8t/r0iaIOn2JK49yd2zNrkwbaNPLrtMuu220PGWLdIu\nu6QYGQAokHyZj7my8Au5adOkSy4J7WGTxD+Ktp45Vq0EkHZJFF2Z0Ldv3/bjPn36tLf79Omj7du3\n93hzbnfXxIkTNX369LTGWWjcfbmUmyuxVZSXqTFO0Zb06JONG6WBA0PHP/qR9MMfpjE6AEUvgHyZ\nj7my8Mc+zJsnTZrUo8q+ZnRl3GGXAFCITjjhhPbhHo8//rj23HNP7bbbbjrxxBP1xz/+UZL017/+\nVe+//74k6ZRTTtH999+v9evXS5Lee+89rV69OpjgC4NLetTMXjCzydm44ZTqYSorLYk5l/Tok9tv\njxRxq1dTxAEoCrmYKwu/Ry4HPwkFgFxy7bXX6qKLLtLIkSO100476a677pIkXXPNNTr33HN1yCGH\n6LjjjtPgwYMlScOHD9d1112nsWPHqrW1VaWlpbrtttv0qW6GrhciM1sgaZ84D13l7g8m+TLHu3uj\nme0l6W9m9rq7L4pzr8mSJktq/170Vq9Gn2zbJu2xh/TRR9L48VJ4KBEAFINczJXmAQ31SUZVVZUv\nWZKVed8AkHbLly/XwQcfHHQYOSfe18XMXnD3hAuC5DMze1zS992924RmZtdK+tDdb+zquqznx4UL\npVNOCR0/80xohUoASBPyZURPcmThD60EACBHmdnOZrZr27GksQotkpIb3KWTTw4VcZWVUnMzRRwA\n5AgKOQAAMsDMxptZg6RjJc03s7rw+Qozezh82d6S/mFmL0t6XtJ8d38kmIg7ePPN0DYCf/+7dPfd\nUkODtEPhz8gAgHzBT2QAADLA3edKmhvn/FpJnw8fvyXpsCyH1r0pU6Qbw6M7N2+Wdt892HgAAJ3Q\nIwcAAELefz+0SNiNN4a273GniAOAHEWPHAAAkGbPli68MHS8cqW0//6BhgMA6BqFHAAAxay5ObSQ\nyYYN0umnSw8/zNY9AJAHGFoJAECx+sc/pB13DBVxTzwh/fWvFHEAkMCqVavaN//ujeuvvz6N0aRY\nyJnZTDN73cyWmtlcMytPcN3pZrbCzFaa2dRU7gkASK9UEtNxxx2X5miQVSecIPXvH9rs+8QTg44G\nAHJaQRVykv4m6VB3HynpDUnTOl5gZiWSbpN0hqThks41s+Ep3hcACk5tfaPGzFiooVPna8yMhaqt\nb8zKfbtKTNu3b+/yuU8//XQmQkK2uEvvvSeVlgYdCQAkLd358uqrr9bPfvaz9vZVV12lW265pdN1\nU6dO1ZNPPqlRo0bp5ptvVktLi6ZMmaIjjzxSI0eO1O233y5Jeuedd3TiiSdq1KhROvTQQ/Xkk09q\n6tSpampq0qhRo3TeeeelFG+blObIufujUc1nJX05zmVHSVoZXmJZZnaPpHGSXkvl3gBQSGrrGzVt\nzjI1NbdIkho3N2nanGWSpJrRlb16zauvvlp77LGHLr/8ckmhxLTXXnvpv/7rv2Kumzp1qpYvX65R\no0Zp4sSJ6t+/v+bMmaMPP/xQLS0tmj9/vsaNG6f3339fzc3Nuu666zRu3DhJ0i677KIPP/xQjz/+\nuK699lrtueeeeuWVV3TEEUfo7rvvljFMryjU1jdqZt0Krd3cpIryMk2pHtbrf7cA0JVM5MuLLrpI\nEyZM0OWXX67W1lbdc889ev755ztdN2PGDN1444166KGHJEmzZs3S7rvvrsWLF+uTTz7RmDFjNHbs\nWM2ZM0fV1dW66qqr1NLSoo8//lgnnHCCbr31Vr300ku9fOedpXOxk4sk3RvnfKWkNVHtBklHp/G+\nCZFYAOSLmXUr2pNSm6bmFs2sW5H1xDR79my9+OKLWrp0qfbYYw9t375dc+fO1W677aaNGzfqmGOO\n0VlnndWpSKuvr9err76qiooKjRkzRk899ZSOP/74XsWO/JGJX6oAIJFM5MshQ4ZowIABqq+v17p1\n6zR69GgNGDCg2+c9+uijWrp0qe6//35J0gcffKA333xTRx55pC666CI1NzerpqZGo0aN6lVc3em2\nkDOzBZL2ifPQVe7+YPiaqyRtl/SHVAMys8mSJkvS4MGDe/06JBYA+WTt5qYenU9GbxOTJJ122mna\nY489JEnuriuvvFKLFi1Snz591NjYqHXr1mmffWJTw1FHHaVBgwZJkkaNGqVVq1ZRyBWBTPxSBQCJ\nZCJfStKkSZM0e/Zsvfvuu7rooouSeo676xe/+IWqq6s7PbZo0SLNnz9fF1xwgb773e/q/PPPTym+\neLqdI+fup7r7oXH+tBVxF0g6U9J57u5xXqJR0n5R7UHhc4nuN8vdq9y9auDAgT16M9G6SiwAkGsq\nyst6dD5ZbYnpd7/7XdKJSZJ23nnn9uM//OEP2rBhg1544QW99NJL2nvvvbV169ZOz+nbt2/7cUlJ\nSbfz61AYMvVLFQDEk6l8OX78eD3yyCNavHhx3MJMknbddVdt2bKlvV1dXa1f/epXam5uliS98cYb\n+uijj7R69Wrtvffe+sY3vqFJkybpxRdflCSVlpa2X5sOqa5aebqk/5Z0lrt/nOCyxZIOMLOhZraj\npHMkzUvlvskgsQDIJ1Oqh6mstCTmXFlpiaZUD0vpdXuTmDr64IMPtNdee6m0tFR///vftXr16pRi\nQmHJ1C9VABBPpvLljjvuqJNOOklf+cpXVFJSEveakSNHqqSkRIcddphuvvlmTZo0ScOHD9fhhx+u\nQw89VJdccom2b9+uxx9/XIcddphGjx6te++9t31u+uTJkzVy5MjcWOxE0q2S+kr6W3iuxLPufqmZ\nVUi6w90/7+7bzewySXWSSiTd6e6vpnjfblWUl6kxTtFGYgGQi9qGoKV7Xm9bYiovL08qMV1wwQXq\n379/zOPnnXeevvjFL2rEiBGqqqrSQQcdlFJMKCxTqofFTGWQ0vNLFQDEk6l82draqmeffVb33Xdf\nwmtKS0u1cOHCmHPXX399p20FJk6cqIkTJ3Z6/g033KAbbrghpTijWfzRkLmhqqrKlyxZ0qvndpwj\nJ4USy/QJIxizDyArli9froMPPjjQGFpbW3X44Yfrvvvu0wEHHBBoLG3ifV3M7AV3rwoopLyTSn7M\nBBYXA5CKoPPla6+9pjPPPFPjx4/XTTfdFFgcUs9yZDpXrcwpmarWASBfRCemXCniUJhqRleSXwHk\nreHDh+utt95qby9btkxf//rXY67p27evnnvuuWyH1qWCLeQkEguA4paviQkAgCCNGDEirfu9ZUpB\nF3IAgIh8SUwAAKB7Ka1aCQDoWi7PQw4CXw8AQDzkh55/DSjkACBD+vXrp02bNpGcwtxdmzZtUr9+\n/YIOBQCQQ8iXvcuRDK0EgAwZNGiQGhoatGHDhqBDyRn9+vXToEGDgg4DAJBDyJchPc2RFHIAkCGl\npaUaOnRo0GEAAJDTyJe9w9BKAAAAAMgzFHIAAAAAkGco5AAAAAAgz1gurw5jZhskrQ46jjTZU9LG\noINII95Pbiu09yMV3nvi/cT6lLsPTFcwhS7H8mOh/VvOBL5G3eNr1D2+Rt0r1K9R3ByZ04VcITGz\nJe5eFXQc6cL7yW2F9n6kwntPvB8UCr733eNr1D2+Rt3ja9S9YvsaMbQSAAAAAPIMhRwAAAAA5BkK\nueyZFXQAacb7yW2F9n6kwntPvB8UCr733eNr1D2+Rt3ja9S9ovoaMUcOAAAAAPIMPXIAAAAAkGco\n5DLEzM42s1fNrNXMEq6eY2arzGyZmb1kZkuyGWNP9OD9nG5mK8xspZlNzWaMPWFme5jZ38zszfDf\n/RNc1xL+3rxkZvOyHWd3uvt6m1lfM7s3/PhzZjYk+1EmL4n3c4GZbYj6nkwKIs5kmdmdZrbezF5J\n8LiZ2c/D73epmR2eXB9drwAABIZJREFU7Rh7Ion38zkz+yDq+3N1tmNE5hVafsuEQsuZmVAoeTgT\nCi23Z0Kh/b7QWxRymfOKpAmSFiVx7UnuPirHl0vt9v2YWYmk2ySdIWm4pHPNbHh2wuuxqZIec/cD\nJD0WbsfTFP7ejHL3s7IXXveS/HpfLOl9d/+MpJsl3ZDdKJPXg38/90Z9T+7IapA9N1vS6V08foak\nA8J/Jkv6VRZiSsVsdf1+JOnJqO/Pj7IQE7Kv0PJbJhRazsyEvM/DmVBouT0TCvT3hV6hkMsQd1/u\n7iuCjiNdknw/R0la6e5vufs2SfdIGpf56HplnKS7wsd3SaoJMJbeSubrHf0+75d0iplZFmPsiXz6\n95MUd18k6b0uLhkn6fce8qykcjPbNzvR9VwS7wdFoNDyWyYUYM7MhELIw5lQaLk9E4r9/047Crng\nuaRHzewFM5scdDApqpS0JqrdED6Xi/Z293fCx+9K2jvBdf3MbImZPWtmuZZkkvl6t1/j7tslfSBp\nQFai67lk//18KTwM8X4z2y87oWVMPv2fSdaxZvaymf3VzA4JOhgEqpDyWyYU4v//niiEPJwJhZbb\nM6EYf1+Ia4egA8hnZrZA0j5xHrrK3R9M8mWOd/dGM9tL0t/M7PXwp95Zl6b3kzO6ej/RDXd3M0u0\nfOunwt+fT0taaGbL3P2f6Y4VSfuLpD+5+ydmdolCn0ieHHBMiHhRof8zH5rZ5yXVKjRsFHmm0PJb\nJhRazswE8jACVBS/L1DIpcDdT03DazSG/15vZnMV6i4OJNGl4f00Sor+xGNQ+Fwguno/ZrbOzPZ1\n93fCQ9nWJ3iNtu/PW2b2uKTRknIlgSTz9W67psHMdpC0u6RN2Qmvx7p9P+4eHfsdkn6ShbgyKaf+\nz6TK3f8ddfywmf3SzPZ0941BxoWeK7T8lgmFljMzoQjycCYUWm7PhGL8fSEuhlYGyMx2NrNd244l\njVVognS+WizpADMbamY7SjpHUq6uMDVP0sTw8URJnT49NbP+ZtY3fLynpDGSXstahN1L5usd/T6/\nLGmh5+7mkd2+nw7zx86StDyL8WXCPEnnW8gxkj6IGmqUd8xsn7Z5GmZ2lEI5pph+uUBYAea3TMin\nnJkJhZCHM6HQcnsmFOPvC/G5O38y8EfSeIXG7H4iaZ2kuvD5CkkPh48/Lenl/9/O/aPUDkQBHP7Z\nvVp3YPUWcCuXYOEmbCxcxWvchL2FWxBsxU7Exj/1W4SNxZ1CUBCUS5zL98FAJmnOOUmYnBAyxkPr\nzzEWj/27+Yz5YfXY+m3Zb85nr/Vfsp6qq2p37F9V52P7oLof5+e+Ol467k/y+FDv6l91NLb/VJfV\nc3Vb7S8d8w/zORv3yl11Xf1dOuYv8rmo/lev4/45rk6qk3F8p/Wft17GNbZaOuYf5nP67vzcVAdL\nx2xs5DrYqvVtqRqN+RRr5oZqtBXr8IZqs1Vr+0I1mup54btjZyQLAADAJHxaCQAAMBmNHAAAwGQ0\ncgAAAJPRyAEAAExGIwcAADAZjRwAAMBkNHIAAACT0cgBAABM5g2mhjYMEeJU0gAAAABJRU5ErkJg\ngg==\n","text/plain":["<Figure size 1080x360 with 2 Axes>"]},"metadata":{"tags":[]}}]},{"cell_type":"markdown","metadata":{"id":"a2oKRRjlXoDA","colab_type":"text"},"source":["## Interpretability"]},{"cell_type":"markdown","metadata":{"id":"83DaPyLrrqmE","colab_type":"text"},"source":["Since we standardized our inputs and outputs, our weights were fit to those standardized values. So we need to unstandardize our weights so we can compare it to our true weight (3.5).\n","\n","Note that both X and y were standardized.\n","\n","$\\hat{y}_{scaled} = b_{scaled} + \\sum_{j=1}^{k}W_{{scaled}_j}x_{{scaled}_j}$\n","* $y_{scaled} = \\frac{\\hat{y} - \\bar{y}}{\\sigma_y}$\n","* $x_{scaled} = \\frac{x_j - \\bar{x}_j}{\\sigma_j}$\n","\n","$\\frac{\\hat{y} - \\bar{y}}{\\sigma_y} = b_{scaled} + \\sum_{j=1}^{k}W_{{scaled}_j}\\frac{x_j - \\bar{x}_j}{\\sigma_j}$\n","\n","$ \\hat{y}_{scaled} = \\frac{\\hat{y}_{unscaled} - \\bar{y}}{\\sigma_y} = {b_{scaled}} + \\sum_{j=1}^{k} {W}_{{scaled}_j} (\\frac{x_j - \\bar{x}_j}{\\sigma_j}) $\n","\n","$\\hat{y}_{unscaled} = b_{scaled}\\sigma_y + \\bar{y} - \\sum_{j=1}^{k} {W}_{{scaled}_j}(\\frac{\\sigma_y}{\\sigma_j})\\bar{x}_j + \\sum_{j=1}^{k}{W}_{{scaled}_j}(\\frac{\\sigma_y}{\\sigma_j})x_j $\n","\n","In the expression above, we can see the expression $\\hat{y}_{unscaled} = W_{unscaled}x + b_{unscaled} $ where\n","\n","* $W_{unscaled} = \\sum_{j=1}^{k}{W}_j(\\frac{\\sigma_y}{\\sigma_j}) $\n","\n","* $b_{unscaled} = b_{scaled}\\sigma_y + \\bar{y} - \\sum_{j=1}^{k} {W}_j(\\frac{\\sigma_y}{\\sigma_j})\\bar{x}_j$"]},{"cell_type":"code","metadata":{"id":"uROsFZxYrZYu","colab_type":"code","outputId":"66e94761-f8dc-486f-85d8-94cce3344dad","executionInfo":{"status":"ok","timestamp":1583941081822,"user_tz":420,"elapsed":214,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":51}},"source":["# Unscaled weights\n","W_unscaled = W * (y_std/X_std)\n","b_unscaled = b * y_std + y_mean - np.sum(W_unscaled*X_mean)\n","print (\"[actual] y = 3.5X + noise\")\n","print (f\"[model] y_hat = {W_unscaled[0][0]:.1f}X + {b_unscaled[0]:.1f}\") "],"execution_count":118,"outputs":[{"output_type":"stream","text":["[actual] y = 3.5X + noise\n","[model] y_hat = 3.4X + 7.8\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"rB7rsEQUDHNc","colab_type":"text"},"source":["# TensorFlow"]},{"cell_type":"markdown","metadata":{"id":"KZi6yi9-DJH6","colab_type":"text"},"source":["We will first implement linear regression using native TensorFlow without the ease of Keras. This will help us understand the hidden operations and also appreciate all the work that Keras does for us."]},{"cell_type":"code","metadata":{"id":"GUrPPwm4Gq68","colab_type":"code","colab":{}},"source":["%tensorflow_version 2.x\n","import tensorflow as tf"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"t-MqkgJ3Gq3x","colab_type":"code","colab":{}},"source":["# Set seed for reproducibility\n","tf.random.set_seed(SEED)"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"zNJFgUY0hcDZ","colab_type":"text"},"source":["## Split data"]},{"cell_type":"markdown","metadata":{"id":"ASzn8LT2Xwb2","colab_type":"text"},"source":["When we're working with TensorFlow we normally use the scikit learn's [splitting functions](https://scikit-learn.org/stable/modules/classes.html#module-sklearn.model_selection) to split our data."]},{"cell_type":"code","metadata":{"id":"WqcmdgEChtXI","colab_type":"code","colab":{}},"source":["from sklearn.model_selection import train_test_split"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"mU9AYWbzh4JQ","colab_type":"code","colab":{}},"source":["TRAIN_SIZE = 0.7\n","VAL_SIZE = 0.15\n","TEST_SIZE = 0.15\n","SHUFFLE = True"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"xtLx53hQhCp7","colab_type":"code","colab":{}},"source":["def train_val_test_split(X, y, val_size, test_size, shuffle):\n","    \"\"\"Split data into train/val/test datasets.\n","    \"\"\"\n","    X_train, X_test, y_train, y_test = train_test_split(\n","        X, y, test_size=test_size, shuffle=shuffle)\n","    X_train, X_val, y_train, y_val = train_test_split(\n","        X_train, y_train, test_size=val_size, shuffle=shuffle)\n","    return X_train, X_val, X_test, y_train, y_val, y_test"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"VZJMdevAO8Cg","colab_type":"text"},"source":["The `train_val_test_split` function essentially splits our data twice. First, we separate out the test set. And then we separate the remaining other set into train and validation sets. "]},{"cell_type":"code","metadata":{"id":"hu0t5R7lhCnU","colab_type":"code","outputId":"a2547970-5fb5-4199-aeca-e0547cb1b9f4","executionInfo":{"status":"ok","timestamp":1583941088702,"user_tz":420,"elapsed":596,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":68}},"source":["# Create data splits\n","X_train, X_val, X_test, y_train, y_val, y_test = train_val_test_split(\n","    X, y, val_size=VAL_SIZE, test_size=TEST_SIZE, shuffle=SHUFFLE)\n","print (f\"X_train: {X_train.shape}, y_train: {y_train.shape}\")\n","print (f\"X_val: {X_val.shape}, y_test: {y_val.shape}\")\n","print (f\"X_test: {X_test.shape}, y_test: {y_test.shape}\")"],"execution_count":124,"outputs":[{"output_type":"stream","text":["X_train: (35, 1), y_train: (35, 1)\n","X_val: (7, 1), y_test: (7, 1)\n","X_test: (8, 1), y_test: (8, 1)\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"8LSpLJkgkVRn","colab_type":"text"},"source":["## Standardize data"]},{"cell_type":"markdown","metadata":{"id":"mM9C5xIeT5C8","colab_type":"text"},"source":["We can also use scikit learn to do [preprocessing and normalization](https://scikit-learn.org/stable/modules/classes.html#module-sklearn.preprocessing). "]},{"cell_type":"code","metadata":{"id":"o-8Ii-JwkVWC","colab_type":"code","colab":{}},"source":["from sklearn.preprocessing import StandardScaler"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"M9FOu9-kkVeh","colab_type":"code","colab":{}},"source":["# Standardize the data (mean=0, std=1) using training data\n","X_scaler = StandardScaler().fit(X_train)\n","y_scaler = StandardScaler().fit(y_train)"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"YNWqEGsYkViI","colab_type":"code","colab":{}},"source":["# Apply scaler on training and test data\n","X_train = X_scaler.transform(X_train)\n","y_train = y_scaler.transform(y_train).ravel().reshape(-1, 1)\n","X_val = X_scaler.transform(X_val)\n","y_val = y_scaler.transform(y_val).ravel().reshape(-1, 1)\n","X_test = X_scaler.transform(X_test)\n","y_test = y_scaler.transform(y_test).ravel().reshape(-1, 1)"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"aSAH9M4UnDAm","colab_type":"code","outputId":"58951c19-8852-4d61-b508-2dbbb59a6120","executionInfo":{"status":"ok","timestamp":1583941091375,"user_tz":420,"elapsed":1037,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":119}},"source":["# Check (means should be ~0 and std should be ~1)\n","print (f\"X_train: mean: {np.mean(X_train, axis=0)[0]:.1f}, std: {np.std(X_train, axis=0)[0]:.1f}\")\n","print (f\"y_train: mean: {np.mean(y_train, axis=0)[0]:.1f}, std: {np.std(y_train, axis=0)[0]:.1f}\")\n","print (f\"X_val: mean: {np.mean(X_val, axis=0)[0]:.1f}, std: {np.std(X_val, axis=0)[0]:.1f}\")\n","print (f\"y_val: mean: {np.mean(y_val, axis=0)[0]:.1f}, std: {np.std(y_val, axis=0)[0]:.1f}\")\n","print (f\"X_test: mean: {np.mean(X_test, axis=0)[0]:.1f}, std: {np.std(X_test, axis=0)[0]:.1f}\")\n","print (f\"y_test: mean: {np.mean(y_test, axis=0)[0]:.1f}, std: {np.std(y_test, axis=0)[0]:.1f}\")"],"execution_count":128,"outputs":[{"output_type":"stream","text":["X_train: mean: -0.0, std: 1.0\n","y_train: mean: 0.0, std: 1.0\n","X_val: mean: 0.1, std: 0.6\n","y_val: mean: 0.1, std: 0.7\n","X_test: mean: -0.3, std: 0.7\n","y_test: mean: -0.3, std: 0.6\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"wshepN4XGoMc","colab_type":"text"},"source":["## Weights"]},{"cell_type":"code","metadata":{"id":"ygovkkxfDSaI","colab_type":"code","colab":{}},"source":["# Weights\n","W = tf.Variable(0.)\n","b = tf.Variable(0.)"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"XeMjtijnHdTa","colab_type":"text"},"source":["## Model"]},{"cell_type":"code","metadata":{"id":"PRQZtvGhDVlo","colab_type":"code","colab":{}},"source":["# Model\n","def model(x):\n","    return x*W + b"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"pwYrU1yBDV4G","colab_type":"code","outputId":"8309b7d7-a115-44cf-87eb-8f14295725ff","executionInfo":{"status":"ok","timestamp":1583941093236,"user_tz":420,"elapsed":269,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":51}},"source":["# Forward pass\n","print (X_train[0])\n","print (model(X_train[0]))"],"execution_count":131,"outputs":[{"output_type":"stream","text":["[0.96849907]\n","tf.Tensor([0.], shape=(1,), dtype=float32)\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"Ht0lvKkiHetb","colab_type":"text"},"source":["## Loss"]},{"cell_type":"code","metadata":{"id":"4E08hXZFDV1i","colab_type":"code","colab":{}},"source":["# Loss\n","def mean_squared_error(y_pred, y_true):\n","    return tf.reduce_mean(tf.square(y_pred-y_true))"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"2YBN9uZ4DVit","colab_type":"code","outputId":"890d5323-9587-42a0-d4b8-be55b4bc9563","executionInfo":{"status":"ok","timestamp":1583941095258,"user_tz":420,"elapsed":666,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["# Sample loss\n","print (f\"loss: {mean_squared_error(model(X_train), y_train)}\")"],"execution_count":133,"outputs":[{"output_type":"stream","text":["loss: 1.0\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"r3Ht5Fe0Hhft","colab_type":"text"},"source":["## Training"]},{"cell_type":"code","metadata":{"id":"ec48EbnnH0n4","colab_type":"code","outputId":"a7f97aca-43c4-4a3f-af07-30b9618fbf5d","executionInfo":{"status":"ok","timestamp":1583941096720,"user_tz":420,"elapsed":808,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":187}},"source":["for epoch in range(NUM_EPOCHS):\n","    with tf.GradientTape() as tape:\n","        # Forward pass\n","        y_pred = model(X_train)\n","\n","        # Loss\n","        train_loss = mean_squared_error(y_pred=y_pred, y_true=y_train)\n","    \n","    # Gradients\n","    gradients = tape.gradient(train_loss, [W, b])\n","\n","    # Update weights\n","    W.assign_sub(gradients[0] * LEARNING_RATE)\n","    b.assign_sub(gradients[1] * LEARNING_RATE)\n","\n","    # Metrics\n","    if (epoch%5) == 0:\n","        print (f\"Epoch: {epoch} | train_loss: {train_loss.numpy():.2f}\")"],"execution_count":134,"outputs":[{"output_type":"stream","text":["Epoch: 0 | train_loss: 1.00\n","Epoch: 5 | train_loss: 0.13\n","Epoch: 10 | train_loss: 0.03\n","Epoch: 15 | train_loss: 0.02\n","Epoch: 20 | train_loss: 0.02\n","Epoch: 25 | train_loss: 0.02\n","Epoch: 30 | train_loss: 0.02\n","Epoch: 35 | train_loss: 0.02\n","Epoch: 40 | train_loss: 0.02\n","Epoch: 45 | train_loss: 0.02\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"zQeIW-cNHliM","colab_type":"text"},"source":["## Evaluation"]},{"cell_type":"code","metadata":{"id":"DeNmcBWcHlcu","colab_type":"code","colab":{}},"source":["# Predictions \n","pred_train = W*X_train + b\n","pred_test = W*X_test + b"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"yescrXo2KIoR","colab_type":"code","outputId":"06ad2fd8-4ea5-4d9f-af78-c8cbf29eaf5b","executionInfo":{"status":"ok","timestamp":1583941098020,"user_tz":420,"elapsed":275,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["# Train and test MSE\n","train_mse = np.mean((y_train - pred_train) ** 2)\n","test_mse = np.mean((y_test - pred_test) ** 2)\n","print (f\"train_MSE: {train_mse:.2f}, test_MSE: {test_mse:.2f}\")"],"execution_count":136,"outputs":[{"output_type":"stream","text":["train_MSE: 0.02, test_MSE: 0.01\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"mJin5A8UKIiI","colab_type":"code","outputId":"6269f3d8-00f5-4b27-c9c7-256f49d71ba8","executionInfo":{"status":"ok","timestamp":1583941099303,"user_tz":420,"elapsed":885,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":336}},"source":["# Figure size\n","plt.figure(figsize=(15,5))\n","\n","# Plot train data\n","plt.subplot(1, 2, 1)\n","plt.title(\"Train\")\n","plt.scatter(X_train, y_train, label='y_train')\n","plt.plot(X_train, model(X_train), color='red', linewidth=1, linestyle='-', label='model')\n","plt.legend(loc='lower right')\n","\n","# Plot test data\n","plt.subplot(1, 2, 2)\n","plt.title(\"Test\")\n","plt.scatter(X_test, y_test, label='y_test')\n","plt.plot(X_test, model(X_test), color='red', linewidth=1, linestyle='-', label='model')\n","plt.legend(loc='lower right')\n","\n","# Show plots\n","plt.show()"],"execution_count":137,"outputs":[{"output_type":"display_data","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAA3IAAAE/CAYAAAADjvF6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOzdeXhU5f3//9ebGCGKGhBcEkSotbiB\nxEZc4ta6xKqVSFvrDrUUa+uv9qdNBe0Hl48KFK22VltpP251o9oYFdS4UFdcQIOCIhUVlMEFgViR\nKFnu7x9nkpkhM8lMZjmzPB/XlctzzpzMeSeIt6+5N3POCQAAAACQO/r4XQAAAAAAIDEEOQAAAADI\nMQQ5AAAAAMgxBDkAAAAAyDEEOQAAAADIMQQ5AAAAAMgxBDkgy5lZkZltMLOhftcCAACA7ECQA1Is\nGLo6vtrNrDns/PRE38851+ac6++c+yAd9QIAkGmpbivD3vclMzsjlbUC2WoLvwsA8o1zrn/HsZmt\nkDTROfdkrPvNbAvnXGsmagMAIBsk2lYC6IoeOSDDzOxKM5ttZveY2ReSzjCzg4KfIjaZ2Udm9icz\nKw7ev4WZOTMbFjy/M/j6o2b2hZm9aGbDffyRAABIqeC0gv8xs/fM7DMzu8vMSoOvbW1m95rZumC7\n+bKZDTCzayXtL+nvwZ69a/39KYD0IsgB/jhJ0t2StpM0W1KrpPMlDZJUJelYSed08/2nSfofSQMl\nfSDpf9NZLAAAGfYbScdIOkTSEEktkq4LvjZR3qiycnnt5nmSNjnnLpS0QF7vXv/gOZC3CHKAP553\nzj3snGt3zjU75xY45152zrU6596TNEvS4d18//3OuYXOuRZJd0kanZGqAQDIjJ9LmuycW+2c+0rS\n5ZJ+bGYmL9QNlrRbsN1c4Jz70s9iAT8wRw7wx4fhJ2a2h6RrJX1b0lby/m6+3M33fxx2vFFS/1g3\nAgCQS4JhbRdJj5iZC3upj6TtJf2fpJ0k3W9m/SXdIel/nHNtGS8W8BE9coA/3GbnN0taIumbzrlt\nJU2VZBmvCgAAnznnnKSApO8650rDvvo55z5zzn3tnJvqnNtD0mGSfiTplI5v96tuINMIckB22EbS\n55K+NLM91f38OAAA8t1fJU03s10kycx2MLPvB4+PMrO9zKyPpP/Km2feHvy+TyR9w4+CgUwjyAHZ\n4UJJ4yV9Ia93bra/5QAA4KvfS3pS0rzgCs/zJe0XfK1c0oPy2swlkh5RqN28TtJZZrbezH6f2ZKB\nzDKv9xoAAAAAkCvokQMAAACAHEOQAwAAAIAcQ5ADAAAAgBxDkAMAAACAHEOQAwAAAIAcs4XfBXRn\n0KBBbtiwYX6XAQBIs1dfffUz59xgv+vIFbSPAFA4YrWRWR3khg0bpoULF/pdBgAgzcxspd815BLa\nRwAoHLHaSIZWAgAAAECOIcgBAAAAQI4hyAEAAABAjiHIAQAAAECOIcgBAAAAQI4hyAEAAABAjiHI\nAQAAAECOIcgBAAAAQI7J6g3BAQDZqb4xoJkNy7S6qVllpSWqrR6hmopyv8sCAKBg0CMHAEhIfWNA\nU+oWK9DULCcp0NSsKXWLVd8Y8Ls0AAD819oqffhh2h9DkAMAJGRmwzI1t7RFXGtuadPMhmU+VQQA\nQJZ4/33psMOkq69O+6NSEuTM7BYz+9TMlsR4/Qgz+9zMFgW/pqbiuQCAzFvd1JzQdQAACsI990gH\nHCD98IfSjTem/XGp6pG7TdKxPdzznHNudPDrihQ9FwCQYWWlJer/9UbNue187fbZhxHXAQAoOF98\nIY0fL11+udTQIF1wgdQn/QMfU/IE59yzktal4r0AANntpvXzteT6k7XPJ++qqWQbSVJJcZFqq0f4\nXBkAABn2yitSRYXUt6/06qvecYZkctXKg8zsdUmrJf3GOfdmBp8NAEjW+vXSwIHaV9Lq71TrR0fX\nat3nX6mcVSsBAIWmrU36/e+l66+XbrpJ+sEPMl5CpoLca5J2dc5tMLPjJNVL2j3ajWY2SdIkSRo6\ndGiGygOAwhT3NgLXXCPV1nrHS5aobO+99UJmS81JZnaLpBMkfeqc2yfK6ybpj5KOk7RR0gTn3GuZ\nrRIAkJBVq6SzzpLa26WFC6VddvGljIysWumc+69zbkPw+BFJxWY2KMa9s5xzlc65ysGDB2eiPAAo\nSHFtI/DZZ5KZF+JOOUVyTtp7b99qzkG3qfs55N+T98Hm7vI+xPxLBmoCAPTWAw9I3/62dOSR0lNP\n+RbipAz1yJnZTpI+cc45MxsjL0CuzcSzAQDRdbeNQE1FubTVVlJzcCXKt9+WRjAHLlHOuWfNbFg3\nt4yVdIdzzkl6ycxKzWxn59xHGSkQABCfjRu9RUyeeEJ68EHpwAP9rig1Qc7M7pF0hKRBZrZK0qWS\niiXJOfdXST+UdK6ZtUpqlnRKsNECAPgk1nYB2/7nLcmODF3gP9fpVC4pfNfYVcFrBDkAyBaLFkmn\nnipVVkqNjdK22/pdkaQUBTnn3Kk9vP5nSX9OxbMAAKlRVlqiwGZhbsWME0Injz8uHX10hqtCLMwh\nB4AMa2+X/vQn6aqrpOuuk844w++KImRkjhwAIPvUVo9QSXGRJGnUR/+JDHHOqX7QXqqaPk/DJ89V\n1fR5kXPnkCoBSeETLIYEr3XBHHIAyKBPPpGOP166917p5ZezLsRJmd1+AACQRTpWp6zZb0jntef+\n/i8d+tNxnQuhdMyh61gIJfz7kBIPSTrPzO6VdICkz5kfBwA+e/RR6ac/9b6mTpWKi/2uKCqCHAAU\nqrvvVs3pp4fOndOhwcMeF0JBXOKYQ/6IvK0HlsvbfuAn/lQKANBXX0mTJ0t1ddI990iHH+53Rd0i\nyAFAITILHdfVSSedFPFyrIVQYl1HdHHMIXeSfpmhcgAAsbz1lnTaadI3v+ktbjJwoN8V9Yg5cgBQ\nSGbNigxxznUJcZK3EEo0sa4DAJCTnJNuvtnrfTvvPOm++3IixEn0yAFA4QgPcA0N0jHHxLy1tnpE\nxBw5SSopLlJtNXvJAQDyxNq10sSJ0sqV0nPPSXvs4XdFCaFHDgDyQH1jIPYKkz//eddeuG5CnOQt\naDJt3EiVl5bIJJWXlmjauJHMjwMA5Id586TRo6XddpNefDHnQpxEjxwA5LxuV5gMW5FSTzwhHXVU\n3O9bU1FOcAMA5JeWFm8lyjvukG69tccPNrMZQQ4AckR9Y0AzG5ZpdVOzykpLVFs9QjUV5VFXmJx5\n31U64crnQxecy3C1AABkmeXLvQVNdthBamz0/pnDCHIAkAO663WLWEnSOa34/fdD5wn2wgEAkHec\n83rgfvMb6dJLpV/+MnLKQY4iyAFADuhuX7ey0hIFmpo1+66LdMCqNztfr5r2lF446ruZLhUAgOzR\n1CSde660eLE3L27kSL8rShmCHADkgO72dbvuR6NUUzm089pJZ1yjt4ftrWmsMAkAKGTz50unny4d\nf7y0YIFUkl9b6BDkACAHdPS6be79GSdIM0Lnwy+ao7LSEk0Lzp8DAKDgtLZKV18t3XSTt3/qiSdm\n7NGx5rOnA0EOAHLA5vu6FbW36d2ZY0M3vPCCdPDBet+n+gAAyAorV0pnnCH17Su99ppUVpaxR3e7\ninQawhxBDgByQEcDMLNhmV6YcmTki6xICQCA9M9/SuedJ9XWShdeKPXJ7JbZ3c1nJ8gBQAGr2WOg\navYLC3GLFkn77utfQQAAZIMNG6Rf/Up6/nnpkUekykpfyuhuPns6ZDamAgB6x0zaaqvQuXOEOAAA\nFiyQKiq8dvK113wLcZI3nz2R68kiyAFANmtqitzr5o03GEoJAEB7u/T733srUl51lfR//yf17+9r\nSbXVI1RSXBRxraS4SLVpWkWaoZUAkK0236yUAAcAgBQISGedJW3a5PXI7bqr3xVJipzPzqqVAOCz\nTC4j3OmTT6Sddgqdv/uu9I1vpPeZAADkggcflM45R/rlL6UpU6QtsivO1FSUZ2z7n+z6yQEgi2R6\nGWFJ9MIBABDNxo3Sb34jPfqoVFcnHXyw3xX5jjlyABBDd8sIp9yKFZEhbvXqlIW4+saAqqbP0/DJ\nc1U1fZ7qGwMpeV8AADLijTek/feX1q/3VmwmxEmiRw4AYsrYMsJp7IXzpVcRAIBUcE664Qbpf/9X\nuvZa6cwzu7aZBYweOQCIIe3LCL/5ZmSDtG5dyodSZrRXEQCAVPn0U+mEE6S77pJeeslb3IQQF4Eg\nBwAxpHUZYTNpn31C585JAwYk/76byfTmpAAAJK2hQRo92vt6/nlpt938rigrMbQSAGJIyzLCCxZI\nY8aEzjdskLbeOslKYysrLVEgSmhL1+akAAD02tdfSxdfLN13n3T33dIRR/hdUVYjyAFAN1K6jLAP\nK1LWVo+ImCMnpXdzUgAAeuXtt6VTT5WGD5caG6Xtt/e7oqzH0EoASLd58yJD3NdfZ2xbgZqKck0b\nN1LlpSUySeWlJZo2biQLnQAAsoNz0t/+Jh16qHTuudK//kWIi1NKeuTM7BZJJ0j61Dm3T5TXTdIf\nJR0naaOkCc6511LxbACIhy8be0tZsS9cJjcnBQAgbuvWST/7mfTuu9Kzz0p77ul3RTklVT1yt0k6\ntpvXvydp9+DXJEl/SdFzAaBHHUvwB5qa5RRagj+t+6ndcUdkiGttZXNvAAA6PP20t5jJrrtKL79M\niOuFlPTIOeeeNbNh3dwyVtIdzjkn6SUzKzWznZ1zH6Xi+QDQne6W4E9LT1UW9MIBAJCVWlqkyy6T\nbr1VuuUW6dju+oLQnUzNkSuX9GHY+argtS7MbJKZLTSzhWvWrMlIcQDyW8aW4P/DHyJDXHs7IQ4A\ngA7vvisdcoi3mEljIyEuSVm32IlzbpZzrtI5Vzl48GC/ywGQB9K+sbfkBbgLLwydO8fGpQAAdPjH\nP6QDD5ROP12aO1facUe/K8p5mQpyAUm7hJ0PCV4DgLRL68bekydHBjbn6IUDAKDD559LZ5whTZsm\nPfmk9Ktf8UFnimRqH7mHJJ1nZvdKOkDS58yPA5ApadnYW2IuHAAA3XnxRa8HrrpaWrhQ2morvyvK\nK6nafuAeSUdIGmRmqyRdKqlYkpxzf5X0iLytB5bL237gJ6l4LgDEK6VL8J9yijR7duicAAcAQEhb\nm9cDd8MN0s03SzU1fleUl1K1auWpPbzuJP0yFc8CAF/RCwcAQGwffOANpdxiC+m116Ry9jFNl6xb\n7AQAstJhhzEXDgCA7tx/v1RZKR1/vPTEE4S4NMvUHDkAyF3hAa6sTAqwVhMAAJ2+/FI6/3zpmWe8\nFSn339/vigoCQQ4AYtltN+m990LnCfTA1TcGUr+4CgAA2ea116RTT5UOPtg73mYbvysqGAytBIBo\nzEIhbv/9Ew5xU+oWK9DULCcp0NSsKXWLVd9ITx4AIE+0t0vXXONt6n355dKttxLiMoweOQAIt8UW\n3mpbHXoxD25mwzI1t7RFXGtuadPMhmX0ygEAct9HH0njx3tDKl95RRo2zO+KChI9cgDQwSwU4k46\nqdeLmaxuak7oOgAAOWPOHGm//byhlM88Q4jzET1yALDZlgL1r61KquesrLREgSihray0pNfvCQCA\nr5qbpd/+Vnr4Yem++6RDDvG7ooJHjxyAwhYW4v5RcZyGXTQn6flstdUjVFJcFHGtpLhItdUjev2e\nAAD4ZskSacwY6dNPpUWLCHFZgh45AIVps164YRfN6TxOdj5bx/exaiUAIKc5J910k3TZZdLMmd68\nuM3aT/iHIAegsDgn9QkNRriu6jT98ZDTutyW7Hy2mopyghsAIHetWSOdfbb08cfS/PnS7rv7XRE2\nw9BKAIXDLCLEyTndf8JPo97KfDakipkda2bLzGy5mU2O8voEM1tjZouCXxP9qBMAOj3xhDR6tLT3\n3tILLxDishRBDkD+a2+PHApy442dK1Iynw3pZGZFkm6U9D1Je0k61cz2inLrbOfc6ODX3zNaJAB0\n2LRJqq2VfvIT6Y47pOnTpS239LsqxMDQSgD5bfOx/JttKZDMfLb6xgDz4NCTMZKWO+fekyQzu1fS\nWElv+VoVAGzuP/+RTj1VGjLEW9Bk0CC/K0IPCHIA8tOmTVLfvqHze++VfvzjqLf2Zj5bfWNAU+oW\nd278HWhq1pS6xZ3vBwSVS/ow7HyVpAOi3PcDMztM0n8k/f/OuQ83v8HMJkmaJElDhw5NQ6kACpJz\n0q23ShddJF1xhfTzn7OgSY4gyAHIOkn3dPXQC5cKMxuWdYa4DsmudomC9bCke5xzX5vZOZJul/Td\nzW9yzs2SNEuSKisrU/8vNYDCs369dM450ttvS08/7c2JQ85gjhyArNLR0xVoapZTqKcrrn3dvvwy\nMsQ1NKQlxEmxV7VMdrVL5J2ApF3CzocEr3Vyzq11zn0dPP27pG9nqDYAhey557wFTXbeWXrlFUJc\nDiLIAcgq3fV0dctM6t8/dO6cdMwxaajQE2tVS1a7xGYWSNrdzIab2ZaSTpH0UPgNZrZz2OmJkpZm\nsD4Ahaa1VZo6VTr5ZG+PuD/+UerXz++q0AsEOQBZJeGerrVrI3vhXnwxbb1w4VjtEvFwzrVKOk9S\ng7yA9k/n3JtmdoWZnRi87Vdm9qaZvS7pV5Im+FMtgLz3/vvSYYdJL78sNTZKxx/vd0VIAnPkAGSV\nstISBaKEtqg9XRmYCxdLMqtdorA45x6R9Mhm16aGHU+RNCXTdQEoMHffLf3619KUKdL550fuq4qc\nRJADkFVqq0dErAYpRenpWrVK2iVs2tGSJb6M7e/NapcAAGTUf/8rnXeeNw+uoUGqqPC7IqQIQQ5A\nVumxp8vHXjgAAHLKyy9Lp50mHXWU9Oqr0tZb+10RUoggByDrRO3pWrZM2mOP0PmKFdKuu2a0LgAA\nckJbmzRjhreQyU03ST/4gd8VIQ0IcgCyH71wAADE58MPpTPP9NrOhQsjpyIgrzDLEUD2WrgwMsR9\n+ikhDgCAWOrqpMpKqbpaevJJQlyeo0cOQHbqRS9cfWOAVSQBAIVn5Upp2DBp8GDp4YelAw7wuyJk\nAD1yALLLvHmRIe7zz+MOcVPqFivQ1CwnKdDUrCl1i1XfGEhfrQAA+O38870QJ3nzyQlxBYMeOQDZ\nI4m5cDMblkVsWSBJzS1tmtmwjF45AED+aW2Vios7T6umPaXVM+YzIqWApKRHzsyONbNlZrbczCZH\neX2Cma0xs0XBr4mpeC6A9KtvDKhq+jwNnzxXVdPnpaeHq64uMsR99VXCc+FWR9lEvLvrAADkrMcf\n7wxxjVNnas/fPcqIlAKUdI+cmRVJulHS0ZJWSVpgZg85597a7NbZzrnzkn0egMzpGK7Y0dPV0ThI\nSt0nfSlakbKstESBKKGtrLSkV+8HAEBW+uY3pXff9Y6/+ELn/fkVNW/W/jEipTCkokdujKTlzrn3\nnHObJN0raWwK3heAz7obrpi0P/whMsS1tia1ImVt9QiVFBdFXCspLlJt9YhevycAAFnjk0+8dvPd\nd71VKZ2T+vdnREoBS0WQK5f0Ydj5quC1zf3AzN4ws/vNjLVQgRyQtsbBTLrwwtC5c1JRUez741BT\nUa5p40aqvLREJqm8tETTxo3k00gAQO674gppp52841dflR57rPOlWCNPGJGS/zK12MnDku5xzn1t\nZudIul3Sd6PdaGaTJE2SpKFDh2aoPADRpHy44m9/K82cGTpvb+86tDIJNRXlBDcAQP5ob4/8oDNK\nu1lbPSJiGoTEiJRCkYoeuYCk8B62IcFrnZxza51zXwdP/y7p27HezDk3yzlX6ZyrHDx4cArKA9Bb\nKR2uaBYZ4pxLaYgDACCvzJ8fCnHXXBOz3WRESuFKRY/cAkm7m9lweQHuFEmnhd9gZjs75z4Knp4o\naWkKngsgzToagaQ22T7zTOnOO0PnScyDAwCgIBx0kPTSS97x2rXSwIHd3s6IlMKUdJBzzrWa2XmS\nGiQVSbrFOfemmV0haaFz7iFJvzKzEyW1SlonaUKyzwWQGUk1DilakRIAgILQ1CQNGOAd77efNx8O\niCEl+8g55x5xzn3LObebc+6q4LWpwRAn59wU59zezrl9nXPfcc69nYrnAshS3/1uZIhzjhAHAEB3\nbrghFOKeeYYQhx5larETAIWCXjgAAOLnnNQnrG+lrS3yHIiBf0sApMbuu9MLBwBAIl5/PRTaLrmk\na6gDukGPHIDk0QsHAEBixo6VHnrIO169Wtp5Z3/rQc4hyAHoveJiqbU1dE6AAwCge19+KfXv7x2X\nl0urVvlbD3IWfbcAescsFOLKywlxAAD05I47QiFuzhxCHJJCjxyAxDCMEgCAxIW3n5s2eaNagCTQ\nIwcgfuGN0MEH9zrE1TcGVDV9noZPnquq6fNU3xhIUYEAAGSZd94JtZ/nnuu1nYQ4pAA9cgB6lsJe\nuPrGgKbULVZzS5skKdDUrCl1iyWp9xuPAwCQjc48U7rzTu/43Xelb3zD33qQVwhyQB6pbwxoZsMy\nrW5qVllpiWqrR8QVjrr9vvAQ98MfSvfdl1SNMxuWdYa4Ds0tbZrZsIwgBwDIDxs3Sltv7R2bSe3t\n/taDvMTQSiBPdPR0BZqa5RTq6epp2GKs75NZ133hkgxxkrS6qTmh6wAA5JSLLw6FuPPOI8QhbeiR\nA/JEb3u6unyfc1p65fdC5+efL11/fcrqLCstUSBKaCsrLUnZMwAA8EX4B6Cffy5tu61/tSDvEeSA\nPNHbnq7w11fMOCHyxTSsSFlbPSJijpwklRQXqbZ6RMqfBQBARixaJFVUhM5Z0RkZwNBKIE/E6tHq\nqaerrLRE5tojQtyMw8eratpTKa2vQ01FuaaNG6ny0hKZpPLSEk0bN5L5cQCA3LTjjqEQ99hjhDhk\nDD1yQJ7obU/XC1OOjDgfdtEclRQXaVoae8hqKsoJbgCA3NbSIm25ZeicAIcMo0cOyBMJ93Rt2hQx\nln/qKZdo+EVz6CEDAKAn118fCnE/+AEhDr6gRw7II3H3dEXZF+4KSVekpSoAAPJIeBv68cfe0ErA\nB/TIAT6pbwyoavo8DZ88V1XT5/W4TUBKfPllZAPEWH4AAOLz7rtdt+UhxMFH9MgBPujYu61jPlvn\n3m1S+oY0RumFAwAAcdh/f2nhQu/47rulU0/t8VvqGwOa2bBMq5uaVVZaotrqEUxbQEoR5AAf9HbP\nt15Zu1YaNCh0/uKL0oEHpvYZAADko/Z2qagodN7WJvXpeUCbLx/YouAwtBLwQW/3fEuYWWSIc44Q\nBwBAPO68MxTiDjzQa0PjCHFS9x/YAqlCjxzgg7LSEgWihLae9nyL26pV0i67hM6XLJH23js17w0A\nQL4Ln47w3nvS8OEJfXvGPrBFQaNHDvBBbfUIlRQXRVyLZ8+3uJhFhjjnCHEAAMTj44+7LmiSYIiT\nYn8wm7IPbAER5ABfJLznWzyWLIlsfFasYEETAADiVVMj7byzd3zDDUm1oWn9wBYIYmgl4JPu9nxL\neKUrVqQEAKB3Np/7tmmTVFyc1Ft2tNmsWol0IsgBWSahla7mz5eqqkLnH30k7bRTpkoFACC3PfKI\ndPzx3vGQIdKHH6bsrbv7wBZIBYZWAlkm7pWuzCJDnHOEOCALmdmxZrbMzJab2eQor/c1s9nB1182\ns2GZrxIoQGahEPfGGykNcUAmEOSALNPjSldz50YOpWxqYiglkKXMrEjSjZK+J2kvSaea2V6b3fZT\nSeudc9+UdJ2kGZmtEigwTU1dFzQZOdK/eoBeIsgBWabbla7MpBNOCF10TtpuuwxVBqAXxkha7px7\nzzm3SdK9ksZuds9YSbcHj++XdKTZ5hNfAaTEuedKAwZ4x1On8kEoclpKghzDRoDUibbS1Y+XPq0X\nphwZutDcTOMD5IZySeHjtVYFr0W9xznXKulzSdtnpDqgkJhJf/2rd7xxo3T55f7WAyQp6cVOwoaN\nHC2vgVpgZg85594Ku61z2IiZnSJv2MiPk302kI82X+nq/RknRN5AgAMKkplNkjRJkoYOHepzNUAO\n2XxhMNpR5IlU9MgxbARIsZqKcr2w5aLIENfaSuMD5J6ApF3CzocEr0W9x8y2kLSdpLWbv5FzbpZz\nrtI5Vzl48OA0lQvkmfCFwZ57jnYUeSUV2w9EGzZyQKx7nHOtZtYxbOSzFDwfyD/sCwfkiwWSdjez\n4fIC2ymSTtvsnockjZf0oqQfSprnHH/pgaQ0N0tbbRU6568U8lDWLXZiZpPMbKGZLVyzZo3f5QCZ\n9dvfRoa49nYaHyCHBee8nSepQdJSSf90zr1pZleY2YnB2/5P0vZmtlzSBZK6zDUHkIDLLw+FuJ/9\njHYUeSsVPXKJDBtZ1d2wEckbOiJpliRVVlbyNw+Fg144IC855x6R9Mhm16aGHX8l6UeZrgvIS+Ft\n6fr1Ummpf7UAaZaKHrnOYSNmtqW8YSMPbXZPx7ARiWEjQKQzz+y6nw1/PQAAiN+SJV3bUkIc8lzS\nPXLBOW8dw0aKJN3SMWxE0kLn3EPyho38IzhsZJ28sAeAXjgAAJIzbJi0cqV3/PDDkfutAnksFUMr\nGTYCJOqUU6TZs0PnBDgAABLT2ioVF4fO29u7fkAK5LGsW+wEyHtmhDgAAJJx002hEHf88V5bSohD\ngUlJjxyAOBx+uPTss6FzAhwAAIkLD2yBgFRW5l8tgI8IckAmhDc6O+8srV7tXy0AAOSiFSuk4cND\n53wgigLH0EognXbbresqWoQ4AAASc8ghoRB3++2EOED0yAHpEx7gxoyRXn7Zv1oAAMhFzkl9wvod\n2toiz4ECxt8EFJT6xoCqps/T8MlzVTV9nuobN9+7PgW23LJrLxwhDgCAxMyeHQpto0d3DXVAgaNH\nDgWjvjGgKXWL1dzSJkkKNDVrSt1iSVJNRXlqHhIe4E46SaqrS837AgBQSMLb03fekb75Tf9qAbIU\nQQ4FY2bDss4Q16G5pU0zG5YlH+RibOxd3xjQzIZlWt3UrLLSEtVWj0hdaAQAIN+sWSPtsEPonLlw\nQEz0T6NgrG5qTuh63MJD3LnnRoS4KXWLFWhqllOoBzAtwzkBAMh1p5wSCnHXXEOIA3pAjxwKRllp\niQJRQltZaUnv3jBGL1yHtBjkfv4AACAASURBVPYAAgCQ5RIalRLepn79tTffHEC36JFDwaitHqGS\n4qKIayXFRaqtHpHYGzkX2eBcemnUTw3T1gMIAECWi3tUyhNPhNrUgQO99pQQB8SFHjkUjI5PAZOa\ns9ZDL1y4lPcAAgCQI+IalRLepr72mlRRkcEKgdxHkENBqako792wxvZ2qSisN+/Pf5Z++ctuv6W2\nekTEKplSL3sAAQDIMd2OSvniC2nbbUMXmQsH9ApDK4GemEWGOOd6DHGSFxqnjRup8tISmaTy0hJN\nGzeS+XEAgLwXa/TJjOduCYW43/6WEAckgR45IJZNm6S+fUPn99zjraiVgF73AAIAkMOijUpZMeOE\n0A0bNkhbb+1DZUD+IMgB0SQwFw4AAEQKn5c+aOnrevCOC0Iv0qYCKUGQA8J9+aXUv3/o/LHHpOpq\n/+oBACBH1VSUq2a/IaEL//63dMQRvtUD5BuCHNCBXjgAAFJj48bIoZO0qUDKsdgJsG5dZIh78UUa\nHAAAeuvHPw6FuCFDaFOBNKFHDoWNXjgAAFInvF1dtUoqZ8EvIF3okUNhCgQiG5slSwhxAIC8V98Y\nUNX0eRo+ea6qps9TfWMgNW/80kuR7apzhDggzeiRQ+GhFw4AUIDqGwMRWwIEmpo1pW6xJCW3VU54\nu3rttdIFF8S+F0DK0COHwrFkSWRjs2IFIQ4AUDBmNiyL2NdNkppb2jSzYVnv3rC1NbJdbW8nxAEZ\nRJBDYTCTRo4MnTsn7bqrf/UAAJBhq5uaE7rerd/8RiouDp0713XEC4C0Ymgl8tv8+VJVVej8o4+k\nnXZK6SPqGwOa2bBMq5uaVVZaotrqEckNUQEAIA3KSksUiBLaykpLEnuj8MC2dKm0xx5JVgagN+iR\nQ/4yiwxxzqUlxE2pW6xAU7OcQvMNUjZ5HACAFKmtHqGS4qKIayXFRaqtHhHfG7z9dtcFTQhxgG8I\ncsg/c+dGNjRNTWmbC5fy+QYAAKRJTUW5po0bqfLSEpmk8tISTRs3Mr5RJGbSnnt6xxdeyBxzIAsw\ntBL5JcMrUqZ0vgEAAGlWU1Ge2PB/56Q+YZ/7t7RIW/C/j0A2SKpHzswGmtkTZvZO8J8DYtzXZmaL\ngl8PJfNMIKp//CMyxDU3Z+TTwljzChKebwAAQLa57rrIEOccIQ7IIskOrZws6Snn3O6SngqeR9Ps\nnBsd/DoxyWcCkcyks84KnTsn9euXkUcnPd8AAIBsZBbaSuCllxhKCWShZIPcWEm3B49vl1ST5PsB\n8bv22sheuNbWjDc0Sc03AAAg26xe3XVBkwMO8K8eADEl2z++o3Puo+Dxx5J2jHFfPzNbKKlV0nTn\nXH2Sz0Why/BcuO4kPN8AAIBstOuu0gcfeMcnnyzNnu1vPQC61WOQM7MnJUVbs/2S8BPnnDOzWP83\nvatzLmBm35A0z8wWO+fejfG8SZImSdLQoUN7Kg+FprZWuuaa0Hl7OxuQAgCQrPC29Msvpa228q8W\nAHHpMcg5546K9ZqZfWJmOzvnPjKznSV9GuM9AsF/vmdmT0uqkBQ1yDnnZkmaJUmVlZUMyEZIFvXC\nAQCQF+6+Wzr99NA5bSuQM5KdI/eQpPHB4/GSHtz8BjMbYGZ9g8eDJFVJeivJ56KQnHlm1/H6NDQA\nACTHLBTi5syhbQVyTLJz5KZL+qeZ/VTSSkknS5KZVUr6uXNuoqQ9Jd1sZu3yguN05xxBDj2qbwyo\nZr8hkRdpZAAASE5TkzQgbMco2lYgJyUV5JxzayUdGeX6QkkTg8fzJY1M5jkoPGv2r1LNwvmd58Mu\nmqOS4iJNawywsAgAAL110UXS73/vHY8ZI738sr/1AOg1dnVE9jHT4LDTYRfNkSQ1t7RpZsMyghwA\nAL0RPk3hv/+VttnGv1oAJI0gh+zxrW9J77zTedoR4MKtbmrOZEUAAOS+xkZpv/1C5wylBPJCsoud\nAKlhFhHiqqY9FfW2stKSTFUEAEkxs4Fm9oSZvRP854AY97WZ2aLg10OZrhN5bvvtQyHu8ccJcUAe\nIcjBX1tuGXVFytrqESopLoq4taS4SLXVIzJcIAD02mRJTznndpf0VPA8mmbn3Ojg14mZKw95bdMm\nr31dt847d046+mh/awKQUgQ5pER9Y0BV0+dp+OS5qpo+T/WNgZ6/yUxqafGOhwyJ+JSwpqJc08aN\nVHlpiUxSeWmJpo0byfw4ALlkrKTbg8e3S6rxsRYUkmuvlfr29Y5PPpleOCBPMUcOSatvDGhK3WI1\nt7RJkgJNzZpSt1iSogevODf2rqkoJ7gByGU7Ouc+Ch5/LGnHGPf1M7OFklrlbdFTn5HqkJ/C29hP\nP5UGD459L4CcRo8ckjazYVlniOvQscJkF+ENzEEH8SkhgJxmZk+a2ZIoX2PD73POOUmx/oO3q3Ou\nUtJpkq43s91iPGuSmS00s4Vr1qxJ7Q+C3PfOO12nKhDigLxGjxySFmslyYjrcfbCAUAucc4dFes1\nM/vEzHZ2zn1kZjtL+jTGewSC/3zPzJ6WVCHp3Sj3zZI0S5IqKyv5jyhC9tvPW5lSkmbP9oZTAsh7\n9MghabFWkuy8Hh7ifvhDQhyAQvGQpPHB4/GSHtz8BjMbYGZ9g8eDJFVJeitjFSK3tbd7bWxHiGtv\nJ8QBBYQgh6TFWmHyhSlHdh3mcd99Ga4OAHwzXdLRZvaOpKOC5zKzSjP7e/CePSUtNLPXJf1b3hw5\nghx6dscdUlGw7a2q8trYzUe/AMhrDK1E0joWJJnZsEyrm5pVtl0/vXBx2GijM86Q/vEPn6oDAH84\n59ZKOjLK9YWSJgaP50sameHSkOvCA9v770vDhvlWCgD/EOSQEp0rTDIXDgCA9Fi9WioPW82ZNhYo\naAytRGq0tUWGuIsuooEBACBVTjghFOJuvJE2FgA9ckiBHnrh6hsDoWGXpSWqrR7B/nAAAMTDOalP\n2OfuLS3SFvzvGwB65JCMr7+ODHF//3vUEDelbrECTc1yCm0WXt8YyGytAADkmjlzQiFu1129NpYQ\nByCI/xqgd+KcC9fdZuH0ygEAEEN4O7tkibT33v7VAiArEeTyVNqGM37xhbTttqHzujrppJNi3h7X\nZuEAAMCzfr00cGDonLlwAGJgaGUOq28MqGr6PA2fPFdV0+d1DldM23BGs8gQ51y3IU6KY7NwAADg\nmTQpFOIuu4wQB6Bb9MjlqI6w1jFssSOsSWkYzvjZZ9LgwaHzp5+WDj88rm+trR4RUafkbRZeWz0i\n8ToAAMhX4UMpN26USvjAE0D3CHI5qruwltLhjEnuC9dls3BWrQQAIOT556VDD/WOzaT2dn/rAZAz\nCHI5qruwVlZaokCU1xMazrhypTRsWOi8sVEaPTrBKj2dm4UDAICQ8A9L58+XDjrIv1oA5BzmyOWo\n7uae1VaPUElxUcT1hIYzmkWGOOd6HeIAAMBmNm6MDHHOEeIAJIwgl6O6C2s1FeWaNm6kyktLZJLK\nS0s0bdzInnvFliyJbFiWL2eiNQAAqTR1qrT11t7xuefSzgLoNYZW5qie5p4lPJwxyblwAACgB+Ft\nbVOTtN12/tUCIOcR5HJYSuaezZ8vVVWFzj/6SNppp+TeEwAAhCxeLI0aFTrnw1IAKUCQK2T0wgEA\nkF5DhkiB4D6uc+dKxx3nbz0A8gZBrhA99ZR01FGhc4Z3AACQWi0t0pZbhs7b27t+gAoASUhqsRMz\n+5GZvWlm7WZW2c19x5rZMjNbbmaTk3kmkmQWGeKcI8QBAJBKN9wQCnE1NV5bS4gDkGLJ9sgtkTRO\n0s2xbjCzIkk3Sjpa0ipJC8zsIefcW0k+G4moq5N+8IPQ+VdfSX37+lcPAAD5KDywMe8cQBolFeSc\nc0slybr/lGmMpOXOufeC994raawkglwc6hsDMVemjBtz4QAASEjC7e/770vf+EbonLYWQJplYh+5\nckkfhp2vCl5DD+obA5pSt1iBpmY5SYGmZk2pW6z6xkB8b/C3v0WGuNZWGhYAAHqQcPt70EGhEHfn\nnbS1ADKixx45M3tSUrRxAZc45x5MdUFmNknSJEkaOnRoqt8+p8xsWKbmlraIa80tbZrZsKznXjl6\n4QAA6JW429/2dqmoKHTe1ib1ycRn5AAQR4+cc+4o59w+Ub7iDXEBSbuEnQ8JXov1vFnOuUrnXOXg\nwYPjfER+Wt3UnNB1SdIf/hAZ4trbu4S4+saAqqbP0/DJc1U1fV78PXwAABSAuNrfe+4JhbjKSq+t\nJcQByKBMbD+wQNLuZjZcXoA7RdJpGXhuzisrLVEgSmNSVloS/Rvi6IXrGC7S8Uljx3ARSclvLg4A\nQB7osf0Nb2+XL5d22y1DlQFASLLbD5xkZqskHSRprpk1BK+XmdkjkuSca5V0nqQGSUsl/dM592Zy\nZReG2uoRKikuirhWUlyk2uoRkTdefHFko+JczKGU3Q0XAQAAsdvf31UO7NreEuIA+CTZVSsfkPRA\nlOurJR0Xdv6IpEeSeVYh6ugh63bVrATnwvVquCYAAAUkWvt73xPXqOzKR70brr9eOv98HysEgMwM\nrUQSairKow95/N3vpKuuCp3HuZhJwsM1AQAoQJ3t7+Zz3zZtkoqL/SsMAIKYlZuLzEIhbvTohFak\njHu4JgAAha6hIRTidtzRa28JcQCyBEEul0ya1HVsfmNjQm9RU1GuaeNGqry0RCapvLRE08aNZKET\nAADC/fjH0rHHeseLFkkff+xvPQCwGYZW5orwAFddLT32WK/fKuZwTQAACl1zs7TVVt5xaam0fr2/\n9QBADPTIZbuxY7v2wiUR4gAAQAz33hsKcQ88QIgDkNXokctm4QHurLOk22/3rxYAAPLZFltIbcHt\neb7+WtpyS3/rAYAe0COXjQ46qGsvHCEOAIDUe/99r81ta5N+8hOvzSXEAcgBBLlsYya99JJ3fMEF\nCa1ICQAAEvCLX0jf+IZ3/J//SLfc4m89AJAAhlZmi113lT74IHROgAMAID3a2ryhlB1ocwHkIHrk\nsoFZKMRdfTUNCgAA6dLYGApxt99OmwsgZxHk/DRqVNe5cFOm+FcPAAD57Oc/l/bbT5J0xP8+quFv\nba+q6fNU3xjwuTAASBxDK/3gnNQnLEPPni2dfLJ/9QAAkM/Wr5cGDpQkvfn/TdYPtztczRu8FSoD\nTc2aUrdYkthjFUBOoUcuQfWNAVVNn6fhk+f27lO8HXeMDHHOEeIAAEiXe+/tDHF6/31NKjtazS1t\nEbc0t7RpZsMyH4oDgN4jyCWgvjGgKXWLFWhqllPoU7y4wpxz3jDKTz/1zh99lHH5AACkS3u7tNde\n0qmnSocf7p0PG6bVTc1Rb491HQCyFUEuATMblvXuU7zRo7v2wh17bBoqBAAAWrpUKiry/jl3rvT0\n051z0stKS6J+S6zrAJCtCHIJSPhTvLY2r+F4/XXvfNEieuEAAEiniy7yeuIkacMG6bjjIl6urR6h\nkuKiiGslxUWqrR6RqQoBICVY7CQBZaUlCkQJbVE/xdtlF2nVqtA5AQ4AgPTZsEHaZhvv+OKLpauu\ninpbx4ImMxuWaXVTs8pKS1RbPYKFTgDkHIJcAmqrR2hK3eKI4ZVdPsXbtEnq2zd0vmyZ9K1vZbBK\nAAAKzMMPSyee6B2//bY0ovvetZqKcoIbgJzH0MoE1FSUa9q4kSovLZFJKi8t0bRxI0ONQd++kSHO\nOUIcABQoM/uRmb1pZu1mVtnNfcea2TIzW25mkzNZY85zTqqq8kLcqFHegiY9hDgAyBf0yCUo6qd4\nGzdKW28dOl+5Uho6NO73rG8MMMQDAPLPEknjJN0c6wYzK5J0o6SjJa2StMDMHnLOvZWZEnPYe+9J\nu+3mHf/zn9KPfuRvPQCQYQS5ZAVXweqU4Fy4ji0NOoZrsjEpAOQH59xSSbLN24lIYyQtd869F7z3\nXkljJRHkunPVVdLvfucdr18vlZb6Ww8A+IChlb31+eeRIe7jj3u1oEmvtzQAAOSDckkfhp2vCl5D\nNM3NXtv7u99Jv/yl1+4S4gAUKHrkeiPJXrhwbEwKALnLzJ6UtFOUly5xzj2Y4mdNkjRJkoYmMHw/\nbzz1lHTUUd7xokXSvvv6Ww8A+Iwgl4imJmnAgNB5CoZzJLSlAQAgqzjnjkryLQKSdgk7HxK8Fu1Z\nsyTNkqTKysrC2tPmhBO8jb2HDvXmxhUV9fw9AJDnGFoZr4EDQyFu330THs5R3xhQ1fR5Gj55rqqm\nz1N9o9dOszEpABS0BZJ2N7PhZralpFMkPeRzTdkjEPBGwcydK91yi7eYGCEOACTRI9ezNWukHXYI\nnTc3S/36JfQW8SxowqqVAJBfzOwkSTdIGixprpktcs5Vm1mZpL87545zzrWa2XmSGiQVSbrFOfem\nj2Vnjz/9STr/fO/400+lwYP9rQcAsgxBrjvhc+GOOEL697979TbdLWjSsZ0BwQ0A8otz7gFJD0S5\nvlrScWHnj0h6JIOlZbeWFm8EzJdfSqefLt15p98VAUBWIshFs26dtP32ofNNm6Ti4l6/HQuaAAAQ\nhxdflA4+2DueP1866CB/6wGALJZUkDOzH0m6TNKeksY45xbGuG+FpC8ktUlqdc5VJvPcVIi5CffR\nR0tPPundNG6c9K9/Jf0sFjQBClNLS4tWrVqlr776yu9Sska/fv00ZMgQFSfx4Rjy1BlnSHfdJfXv\n732gyr8jQMGgvfQk2kYm2yO3RNI4STfHce93nHOfJfm8lIg2Z23mP55TzX6nhm5qa5P6pGYtmNrq\nERHPk1jQBCgEq1at0jbbbKNhw4b1tCl0QXDOae3atVq1apWGDx/udznIFuFz0f/4R+lXv/K3HgAZ\nR3vZuzYyqaTinFvqnMu5nas3n7P2wB0X6oXrgiHuxhu9FSlTFOIkb0GTaeNGqry0RCapvLRE08aN\nZF4ckOe++uorbb/99gXbKG3OzLT99tsX/CeuCHPrraEQt2oVIQ4oULSXvWsjMzVHzkl63MycpJuD\ne+H4pmNu2o5ffKaXb5rQeX34bx/W+784IfawyySwoAlQmAq5UYqG3wckeaNehg+XPvxQOv54ac4c\nvysC4DPah8R/Bz12O5nZk2a2JMrX2ASec4hzbj9J35P0SzM7rJvnTTKzhWa2cM2aNQk8In5lpSU6\n87U5nSHuN8f9WsMumqOyAVt1DrsMNDXLKbRVQMe+bwBQqIYNG6bPPut+hHw896DAvf66tMUWXoh7\n6ilCHIC8ksm2ssceOefcUck+xDkXCP7zUzN7QNIYSc/GuHeWpFmSVFlZ6ZJ9dhdNTXphypGSpCe+\neYB+Nu53klnnnLWetgoAAAC9dN553hQGqVf7sgIAQlI3ESwGM9vazLbpOJZ0jLxFUjLv2mu9vWkk\nPfXPJ3XZT6+WmUXMWWOrAAD5ZMWKFdpjjz00YcIEfetb39Lpp5+uJ598UlVVVdp99931yiuvaN26\ndaqpqdGoUaN04IEH6o033pAkrV27Vsccc4z23ntvTZw4Uc6FPlu78847NWbMGI0ePVrnnHOO2tra\nYpUASE1N3t6sN94oXXmlNxedEAcgS+RqW5lUkDOzk8xslaSDJM01s4bg9TIz69jcdEdJz5vZ65Je\nkTTXOfdYMs9N2Nq1XgPym99IP/6x5JyO/NGRemHyd/X+9OP1wuTvdva2xdoSgK0CAOSq5cuX68IL\nL9Tbb7+tt99+W3fffbeef/55XXPNNbr66qt16aWXqqKiQm+88YauvvpqnXXWWZKkyy+/XIcccoje\nfPNNnXTSSfrggw8kSUuXLtXs2bP1wgsvaNGiRSoqKtJdd93l54+IbHbffZ0fourdd6VLLvG3HgCI\nIhfbyqQWO3HOPSDpgSjXV0s6Lnj8nqR9k3lOUv7yF+kXv/CO335bGtH9kv9sFQAgbdIxkdv1PAJ9\n+PDhGjlypCRp77331pFHHikz08iRI7VixQqtXLlS/wrumfnd735Xa9eu1X//+189++yzqqurkyQd\nf/zxGtAxouGpp/Tqq69q//33lyQ1Nzdrh46VB4EOzkn77istXixVVUnPPZeevwMA8o8P7WUutpWZ\nWrXSP++8I02Y4C1xHIeOnrlUr1oJAPGErnTo27dv53GfPn06z/v06aPW1taEN+d2zmn8+PGaNm1a\nSutEHlm2TNpjD+/4oYek73/f33oA5BYf2stcbCvTPkfOd3/4Q9whrkNNRXnUYZcAkI8OPfTQzuEe\nTz/9tAYNGqRtt91Whx12mO6++25J0qOPPqr169dLko488kjdf//9+vTTTyVJ69at08qVK/0pHtnn\n4otDIe6LLwhxAPJCNraV+d8jBwDo1mWXXaazzz5bo0aN0lZbbaXbb79dknTppZfq1FNP1d57762D\nDz5YQ4cOlSTttddeuvLKK3XMMceovb1dxcXFuvHGG7Xrrrv6+WPAb19+KfXv7x3/9rfSjBn+1gMA\nKZSNbaU5n4b6xKOystItXLjQ7zIAoFeWLl2qPffc0+8ysk6034uZveqcq/SppJyTivaxvjGQumkE\njz4qHXecd/zmm9JeeyVVG4DCQnsZkkgbSY8cAAAFpr4xELGwV6CpWVPqFktSYmHOOek735GeeUba\nc09pyRKpT/7P2gCAbMB/bQEAKDAzG5ZFrM4sSc0tbZrZsCz+N1mxwgttzzwj3XOP9NZbhDgAyCD+\niwsAQIFZ3dSc0PUuZsyQhg/3jtetk045JUWVAQDixdBKAAAKTFlpiQJRQltZaUn33/j111K/ft7x\nOedIf/1rGqoDAMSDHjkAAApMbfUIlRQXRVwrKS5SbfWI2N/0zDOhEPfqq4Q4APAZPXIAABSYjgVN\n4l61sqZGevBBaeedpQ8/lIqKot8HAMgYghwAAAWopqK85xUqP/pIKivzjv/2N2nixPQXBgBZasWK\nFZo/f75OO+20Xn3/1VdfrYsvvjhl9TC0EgAK3IoVK3T33Xf36nsPPvjgFFeDrPGXv4RC3McfE+IA\nFLxk2kvJC3KpRJADgCxR3xhQ1fR5Gj55rqqmz1N9YyAjz+2uYWptbe32e+fPn5+OkuCn1lZp4EDp\nF7+QTj7Z2ytuxx39rgoAOqW6vZw6daquv/76zvNLLrlEf/zjH7vcN3nyZD333HMaPXq0rrvuOrW1\ntam2tlb777+/Ro0apZtvvlmS9NFHH+mwww7T6NGjtc8+++i5557T5MmT1dzcrNGjR+v0009Pqt4O\nDK0EgCyQsg2aw0ydOlUDBw7Ur3/9a0lew7TDDjvo/PPPj7hv8uTJWrp0qUaPHq3x48drwIABqqur\n04YNG9TW1qa5c+dq7NixWr9+vVpaWnTllVdq7NixkqT+/ftrw4YNevrpp3XZZZdp0KBBWrJkib79\n7W/rzjvvlJn19lcCP7zyinTAAd7x889LVVX+1gMAm0lHe3n22Wdr3Lhx+vWvf6329nbde++9euWV\nV7rcN336dF1zzTWaM2eOJGnWrFnabrvttGDBAn399deqqqrSMccco7q6OlVXV+uSSy5RW1ubNm7c\nqEMPPVR//vOftWjRol7+5F3ldZCrbwzEP5EbAHzU3QbNmW6YbrvtNr322mt64403NHDgQLW2tuqB\nBx7Qtttuq88++0wHHnigTjzxxC4hrbGxUW+++abKyspUVVWlF154QYccckivaocPJkyQbr9d6ttX\n+u9/pS239LsiAOgiHe3lsGHDtP3226uxsVGffPKJKioqtP322/f4fY8//rjeeOMN3X///ZKkzz//\nXO+88472339/nX322WppaVFNTY1Gjx7dq7p6krdBLh1pHQDSJekNmqPobcMkSUcffbQGDhwoSXLO\n6eKLL9azzz6rPn36KBAI6JNPPtFOO+0U8T1jxozRkCFDJEmjR4/WihUrCHK5YO1aadAg7/jaa6UL\nLvC3HgDoRjraS0maOHGibrvtNn388cc6++yz4/oe55xuuOEGVVdXd3nt2Wef1dy5czVhwgRdcMEF\nOuuss5KqL5q8nSPXXVoHgGwTayPmHjdo7kFHw3TrrbfG3TBJ0tZbb915fNddd2nNmjV69dVXtWjR\nIu2444766quvunxP3759O4+Liop6nF+HLHDHHaEQ98EHhDgAWS9d7eVJJ52kxx57TAsWLIgazCRp\nm2220RdffNF5Xl1drb/85S9qaWmRJP3nP//Rl19+qZUrV2rHHXfUz372M02cOFGvvfaaJKm4uLjz\n3lTI2yCXrrQOAOnQqw2a49Cbhmlzn3/+uXbYYQcVFxfr3//+t1auXJlUTcgi48dL1dXegia77OJ3\nNQDQo3S1l1tuuaW+853v6OSTT1ZRjL0yR40apaKiIu2777667rrrNHHiRO21117ab7/9tM8+++ic\nc85Ra2urnn76ae27776qqKjQ7NmzO+emT5o0SaNGjWKxk56UlZYoECW0JZvWASAdEt6gOU4dDVNp\naWlcDdOECRM0YMCAiNdPP/10ff/739fIkSNVWVmpPfbYI6makEWc87sCAEhIutrL9vZ2vfTSS7rv\nvvti3lNcXKx58+ZFXLv66qu7bCswfvx4jR8/vsv3z5gxQzNmzEiqznB5G+Rqq0dEzJGTUpPWASBd\n4tqgOUG9bZgmTJjQeTxo0CC9+OKLUb93w4YNkqQjjjhCRxxxROf1P//5z70vGgCAbqS6vXzrrbd0\nwgkn6KSTTtLuu++esvdNt7wNculK6wCQK3K1YQIAIJP22msvvffee53nixcv1plnnhlxT9++ffXy\nyy9nurRu5W2Qk9Lz6TYA5IpcbZgAAPDTyJEjU7rfW7rkdZADAITkSsMEAAB6lrerVgJANnAsJhGB\n3wcAIBrah8R/BwQ5AEiTfv36ae3atTROQc45rV27Vv369fO7FABAFqG97F0bydBKAEiTIUOGaNWq\nVVqzZo3fpWSNfv36aciQIX6XAQDIIrSXnkTbSIIcAKRJcXGxhg8f7ncZAABkNdrL3mFoJQAAAADk\nGIIcAAAAAOQYghwAvdQhbAAABc5JREFUAAAA5BjL5tVhzGyNpJUZfOQgSZ9l8HmZlK8/W77+XBI/\nW67iZ+udXZ1zg9P03nnHh/YxW+Xz37dk8HuJjd9NbPxuYvP7dxO1jczqIJdpZrbQOVfpdx3pkK8/\nW77+XBI/W67iZwMyh38no+P3Ehu/m9j43cSWrb8bhlYCAAAAQI4hyAEAAABAjiHIRZrldwFplK8/\nW77+XBI/W67iZwMyh38no+P3Ehu/m9j43cSWlb8b5sgBAAAAQI6hRw4AAAAAckxBBzkz+5GZvWlm\n7WYWcyUaM1thZovNbJGZLcxkjb2RwM91rJktM7PlZjY5kzX2lpkNNLMnzOyd4D8HxLivLfjntcjM\nHsp0nYno6c/BzPqa2ezg6y+b2bDMV9k7cfxsE8xsTdif1UQ/6kyUmd1iZp+a2ZIYr5uZ/Sn4c79h\nZvtlusbeiuNnO8LMPg/7M5ua6RpRuPK5fUtGPraNycrntjVZ+do2JysX2/aCDnKSlkgaJ+nZOO79\njnNudDYuPRpFjz+XmRVJulHS9yTtJelUM9srM+UlZbKkp5xzu0t6KngeTXPwz2u0c+7EzJWXmDj/\nHH4qab1z7puSrpM0I7NV9k4C/47NDvuz+ntGi+y92yQd283r35O0e/BrkqS/ZKCmVLlN3f9skvRc\n2J/ZFRmoCeiQz+1bMvKqbUxWPretycrztjlZtynH2vaCDnLOuaXOuWV+15Fqcf5cYyQtd86955zb\nJOleSWPTX13Sxkq6PXh8u6QaH2tJhXj+HMJ/5vslHWlmlsEaeytX/x3rkXPuWUnrurllrKQ7nOcl\nSaVmtnNmqktOHD8b4Js8b9+SkW9tY7LyuW1NViH+/YhLLrbtBR3kEuAkPW5mr5r9v/bunkWKLArj\n+P8BcYVFxFXWd8QBQTASZFlfIhEDA0E0MNJAcA38BmYmgh/AyNzAhd1VUERRQ1ERhsEXfEt0GBU3\nUExE4RjUHWnUmWm5ZVXd6ucHxdyuLmrOqdvVZ2717RodaTuYmqwCng88fpHWdd2yiJhK7ZfAshm2\nWyDpjqSbkrpc0Ibphy/bRMQn4C2wpJHo8gz7GtuXpij8LWlNM6H9dKWeX8PaImlc0iVJG9sOxuwr\nfT//vqdvtTFXn2trrlGuzbk6994yr81f3gRJV4Hl33nqeET8N+RutkfEpKTfgSuSHqZRe2tqyquT\nZstt8EFEhKSZbru6NvXZGHBN0kREPK07Vst2ATgbER8k/UV1dXRHyzHZ7O5SnV/vJe0G/qWaZmJW\niz7XtxyujdYg1+ZC9H4gFxE7a9jHZPr5WtI/VB9LtzqQqyGvSWDwCsvqtK51s+Um6ZWkFRExlT7O\nfj3DPqb77JmkG8AmoIvFaph+mN7mhaR5wCLg/2bCyzJnbhExmMcZ4FQDcTWhs+dXroh4N9C+KOm0\npKUR8abNuKw/+lzfcoxYbczV59qaa5Rrc67Ovbd4auUcJP0qaeF0G9hF9WXr0t0G1ktaJ2k+cAAo\n4Q5W54FDqX0I+ObqrKTFkn5J7aXANuB+YxH+mGH6YTDn/cC1KOMfQM6Z21dzy/cADxqM72c6DxxM\nd7j6E3g7MO2paJKWT3+PRNIfVHVkFP74sXKUWt9y9K025upzbc01yrU5V/dqe0SM7ALspZrf+gF4\nBVxO61cCF1N7DBhPyz2qqR2tx56bV3q8G3hEdTWu83mlmJdQ3ZHrMXAV+C2t3wycSe2twETqswng\ncNtxz5HTN/0AnAD2pPYC4BzwBLgFjLUdc425nUzn1ThwHdjQdsxD5nUWmAI+pnPtMHAUOJqeF9Vd\nwZ6m1+DmtmOuMbdjA312E9jadsxeRmfpc33LPC69q401HJPe1tYGjk2RtbmG41JcbVcKzMzMzMzM\nzArhqZVmZmZmZmaF8UDOzMzMzMysMB7ImZmZmZmZFcYDOTMzMzMzs8J4IGdmZmZmZlYYD+TMzMzM\nzMwK44GcmZmZmZlZYTyQMzMzMzMzK8xnDMcwWdsKVrgAAAAASUVORK5CYII=\n","text/plain":["<Figure size 1080x360 with 2 Axes>"]},"metadata":{"tags":[]}}]},{"cell_type":"markdown","metadata":{"id":"62hMRBHIHr5m","colab_type":"text"},"source":["## Interpretability"]},{"cell_type":"code","metadata":{"id":"YIsj9yiRHthX","colab_type":"code","outputId":"1581af40-96a5-4ada-e5d2-da5a4d7a6a1c","executionInfo":{"status":"ok","timestamp":1583941101022,"user_tz":420,"elapsed":458,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":51}},"source":["# Unscaled weights\n","W_unscaled = W.numpy() * (y_std/X_std)\n","b_unscaled = b.numpy() * y_std + y_mean - np.sum(W_unscaled*X_mean)\n","print (\"[actual] y = 3.5X + noise\")\n","print (f\"[model] y_hat = {W_unscaled:.1f}X + {b_unscaled:.1f}\") "],"execution_count":138,"outputs":[{"output_type":"stream","text":["[actual] y = 3.5X + noise\n","[model] y_hat = 3.4X + 7.6\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"6nAJqYIwrxLW","colab_type":"text"},"source":["# TensorFlow + Keras"]},{"cell_type":"markdown","metadata":{"id":"FP6JKnmtrzIK","colab_type":"text"},"source":["Now that we've implemented linear regression with Numpy and native TensorFlow, let's do the same with TensorFlow + Keras."]},{"cell_type":"markdown","metadata":{"id":"X3u7SbMfbqd9","colab_type":"text"},"source":["## Weights"]},{"cell_type":"markdown","metadata":{"id":"S7FufaY1bzZA","colab_type":"text"},"source":["We will be using [Dense layers](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/layers/Dense) in our MLP implementation. The layer applies an activation function on the dot product of the layer's inputs and its weights.\n","\n","$ z = \\text{activation}(XW)$ "]},{"cell_type":"code","metadata":{"id":"WLyU8Y5ZrZl5","colab_type":"code","colab":{}},"source":["from tensorflow.keras.layers import Dense\n","from tensorflow.keras.layers import Input"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"7c-gJ6BgrZjk","colab_type":"code","outputId":"54312021-f56b-4bd4-c39e-ab6ce411f4b7","executionInfo":{"status":"ok","timestamp":1583941108261,"user_tz":420,"elapsed":392,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["x = Input(shape=(INPUT_DIM,))\n","fc = Dense(units=OUTPUT_DIM, activation='linear')\n","z = fc(x)\n","W, b = fc.weights\n","print (f\"z {z.shape} = x {x.shape} · W {W.shape} + b {b.shape}\")"],"execution_count":140,"outputs":[{"output_type":"stream","text":["z (None, 1) = x (None, 1) · W (1, 1) + b (1,)\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"t2j5_5SopsRF"},"source":["## Model"]},{"cell_type":"markdown","metadata":{"id":"phjJCXREb5hG","colab_type":"text"},"source":["Our goal is to learn a linear model $\\hat{y}$ that models $y$ given $X$. \n","\n","$\\hat{y} = XW + b$\n","* $\\hat{y}$ = predictions | $\\in \\mathbb{R}^{NX1}$ ($N$ is the number of samples)\n","* $X$ = inputs | $\\in \\mathbb{R}^{NXD}$ ($D$ is the number of features)\n","* $W$ = weights | $\\in \\mathbb{R}^{DX1}$ \n","* $b$ = bias | $\\in \\mathbb{R}^{1}$ "]},{"cell_type":"code","metadata":{"id":"cBQtSQIorZiB","colab_type":"code","colab":{}},"source":["from tensorflow.keras.models import Model\n","from tensorflow.keras.utils import plot_model"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"ywbqbZwOVGgf","colab_type":"code","colab":{}},"source":["class LinearRegression(Model):\n","    def __init__(self, output_dim):\n","        super(LinearRegression, self).__init__(name='linear_regression')\n","        self.fc1 = Dense(units=output_dim, activation='linear', name='W')\n","    \n","    def call(self, x_in, training=False):\n","        y_pred = self.fc1(x_in)\n","        return y_pred\n","    \n","    def summary(self, input_shape):\n","        x_in = Input(shape=input_shape, name='X')\n","        summary = Model(inputs=x_in, outputs=self.call(x_in), name=self.name)\n","        return plot_model(summary, show_shapes=True) # forward pass"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"0U7wD5a6VSZ8","colab_type":"code","outputId":"eae68266-aadb-47e7-c16d-9e554f43a910","executionInfo":{"status":"ok","timestamp":1583941112893,"user_tz":420,"elapsed":544,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":201}},"source":["# Initialize model\n","model = LinearRegression(output_dim=OUTPUT_DIM)\n","model.summary(input_shape=(INPUT_DIM,))"],"execution_count":143,"outputs":[{"output_type":"execute_result","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAASUAAAC4CAIAAAAqim0JAAAABmJLR0QA/wD/AP+gvaeTAAAePklE\nQVR4nO3dezxU6/448GcxMUbjVkikXNJOoZuOEXU62lSK3DKls4961XHZ56Ccs0W1k6LaeuFITq86\nqr27UfRCRe2tV3Y5O3YlEju3tkoqJJcYjLF+fzzf73znh2qMmTUXn/dfrWeteZ5nzfi0bs+zPgRJ\nkggAQAklaXcAgHEE4g0A6kC8AUAdiDcAqEMTXLh3715CQoK0ugKA4mGxWDt27OAv/n/Ht5cvX2Zm\nZlLeJTBUcXFxcXGxtHshcY2NjYr991ZcXHzv3j3BEtrwjS5fvkxVf8DIfHx80Dj4IS5duuTr66vA\nu4l/R0Fw/QYAdSDeAKAOxBsA1IF4A4A6EG8AUAfiTXHk5eVpampevXpV2h0Rs8DAQOJ/bdq0SXBV\nQUFBZGTk4OCgh4eHsbExnU43NDR0d3d//Pix8PUPDg4mJiba29sLFubm5h4+fJjH4/FLsrOz+d2Y\nPHmyaPsC8aY4FHiqh46OTn5+fnV1dVpaGr9w7969ycnJUVFRg4ODd+/evXDhQltbW1FREYfDWbp0\naVNTkzA119bWLl26dMeOHT09PYLlbm5udDrdycmpvb0dl7i7uzc2Nt65c2f16tUi7wjEm+JwdXXt\n6OhYu3atpBvicDhDjgaSpqamtnLlSgsLC1VVVVxy6NCh9PT0S5cuMZlMhBCLxXJwcGAwGCYmJrGx\nsR0dHWfOnPlsteXl5Tt37gwKCpo3b97wtaGhoTY2NqtXrx4YGEAIEQRhaGjo6Og4c+ZMkXcE4g2M\nWlpaWnNzsxQ7UFdXt2fPnn379tHpdIQQjUYTPIs2NTVFCNXX13+2Hhsbm6ysLD8/P34YDxEdHV1W\nVpaUlCSmjkO8KYqioiJjY2OCIFJSUhBCqamp6urqDAYjJydn1apVGhoaRkZGFy9exBsnJyfT6XQ9\nPb3AwEADAwM6nW5vb19SUoLXhoSEqKioTJkyBS9+/fXX6urqBEG0trYihMLCwsLDw+vr6wmCMDc3\nRwjduHFDQ0MjNjaWsp1NTk4mSdLNzW3EtRwOByGkoaEx9oa0tbWXLVuWlJQkrnN1iDcF4eDg8Msv\nv/AXg4ODt2/fzuFwmExmRkZGfX29qanptm3buFwuQigkJMTf37+npyc0NLShoaG0tHRgYODLL798\n+fIlQig5OXn9+vX8qo4dO7Zv3z7+YlJS0tq1a83MzEiSrKurQwjhmwqDg4OU7ez169dnzZrFYDBG\nXPvrr78ihBwcHMTS1vz581+9elVeXi6W2iDeFJy9vb2Ghoauri6bze7u7n7x4gV/FY1Gmz17tqqq\nqqWlZWpqaldX1+nTp0VowtXVtbOzc8+ePeLr9ad0d3f//vvvZmZmw1e9ffs2PT09NDSUxWJ97Og3\nWvhqraKiQiy1jTBeGSgkFRUVhBA+vg23aNEiBoPx9OlTajsliubmZpIkRzy4sVis7u7u9evXHzhw\nYMKECWJpDjf09u1bsdQG8Qb+h6qqaktLi7R78Xm9vb0IoRHvcOjp6aWlpc2ZM0eMzampqfEbHTs4\nnwQIIcTlctvb242MjKTdkc/DASD4JJpPV1dXS0tLvM319/fzGx07OL4BhBAqLCwkSdLOzg4v0mi0\nj515Sp2enh5BEB0dHcNXSWJsDW5IX19fLLXB8W38GhwcfP/+/cDAwOPHj8PCwoyNjf39/fEqc3Pz\ntra27OxsLpfb0tLy/PlzwQ/q6Og0NTU1NDR0dXVxudz8/HwqnwcwGAxTU9PGxsYh5XV1dfr6+r6+\nvoKFbDZbX1+/tLRU5OZwQ1ZWViLXIAjiTUGkpKTY2toihCIiItzd3VNTUxMTExFC1tbWz549O3ny\nZHh4OEJo5cqVtbW1+CO9vb1WVlZqamqOjo4WFha3b9/mXxQFBwcvX758w4YNs2bN2r9/Pz6bYrFY\n+IFBUFCQnp6epaXl6tWr29raqN9ZV1fXyspK/JyNb8RHZP39/c3NzTk5OSPWU1xc7ODgMHXq1JKS\nkvLycgMDgyVLlty5c0dwm/v37xsaGlpbW4un66SAjIyMISVAKry9vb29vSXaREBAgI6OjkSb+Cwh\n/94CAgIMDQ0FS2pra2k02tmzZz/7WR6P5+jomJaWJloPW1tb6XT6kSNHBAtDQ0MnTZokzMeH/45w\nfBu/RrzlIJs4HM7Nmzdra2vx3Qtzc/OYmJiYmJgPHz584lM8Hi87O7urq4vNZovWbnR09Lx580JC\nQhBCJEk2NTUVFRXhp/yigXgDcqCtrQ2PV96yZQsuiYyM9PHxYbPZI944wQoLC7OysvLz8z82EuXT\nEhISysrK8vLy8KO8nJwcPF75+vXrou0FEi3e7O3tJ0yYQBCEiorKggUL3rx5gxA6c+aMkZERnhqU\nmpr62UqKi4tnz56tpKREEIS+vv6BAwdE6IlosrKyTE1N8USmKVOmDJlSNR5ERUWdPn26o6PDxMRE\n9t9Id/z4cf752Llz5/jlsbGxISEhBw8e/NgHnZyczp8/zx8IOio5OTl9fX2FhYXa2tq4ZN26dYLn\nmSLUiZCo128FBQV4uGp3dze/8MKFC4sXL+7v7xemBszFxQUh9P79e+E/Ii5mZmaamprUtysMCq7f\nZIHC3y8Q2/Wbk5NTYGBgXV3dzp07cUlNTU1ERERGRoa4xtGIF/VTtgAYTvTrt/j4eFNT05SUlMLC\nQg6H4+Pjc/To0RkzZoivb+Ik9SlbAKCxxJu6ujoeTr5ly5a//vWvTk5O7u7uQ7YZ1cwoKqdsCePu\n3buWlpaampp0Ot3KyurmzZsIoa1bt+ILPzMzs0ePHiGENm/ezGAwNDU1c3NzEUI8Hu/bb781NjZW\nU1OztrbGp0zfffcdg8FgMpnNzc3h4eGGhobV1dVCdgMoFMGTSxHOp0NDQxFCJiYmI162Xbt2jclk\nxsTEfOzjQ67fdu3ahRC6detWR0dHc3Ozo6Ojuro6v+aAgAB1dfWqqqre3t7KykpbW1smk/nixQu8\n1s/PT19fn19zfHw8QqilpQUvenl54SlbfJ+9frt8+XJ0dHRbW9u7d+/s7Oz4j1y8vLyUlZVfvXrF\n33Ljxo25ubn43//4xz9UVVUzMzPfv38fFRWlpKR0//59/q6FhoYePXrU09Pzt99++0TTcP2mGMT/\n/A3fn21oaLh79+7wtaLNjKJgypYwvL299+7dq62traOj4+bm9u7dOzx8PigoiMfj8dvt7Oy8f/8+\nfodMb29vamqqh4eHl5eXlpbW7t27J0yYINjDQ4cO/e1vf8vKyvriiy8k1G0gy8YUb319fZs3b96z\nZ4+SktKWLVu6urrE1S1MdqZs4ZtA+AHxn/70JwsLi1OnTpEkiRBKT09ns9nKysoIoerq6p6enrlz\n5+JPqampTZkyRbQeZmZmEooOj3WUdi8kaPizljHND9i+ffuyZctiYmL6+/sPHz4cHh5+4sSJsVQ4\nWhKdsnX9+vX4+PjKysrOzk7BmCcIIjAwcMeOHbdu3VqxYsUPP/xw/vx5vKq7uxshtHv37t27d/O3\nNzAwEKF1Ozu77du3j20PZN29e/eSkpLwWaVCwkNYBYkeb5cuXXr48GFRURFCaN++fdeuXTt58qSn\np+fKlSvH1EehSWLK1p07dx4+fLh9+/YXL154eHh4enqeOnVq6tSpR48e/eabb/ib+fv7R0VF/ec/\n/5k2bZqGhsb06dNxua6uLkIoMTExLCxsjD0xMjISfImIokpKSlLg3RyeakvEeKuvr//mm28KCwvx\niZaqqur3339vZ2e3devWJ0+eiH3O34gkMWXr4cOH6urqCKGKigoulxscHIxfrkYQhOBm2travr6+\n6enpTCZz27Zt/PJp06bR6fSysrIxdgMoKlGu3/r6+nx9fZOSkgSfti1cuDAyMvLVq1f4jiUm9plR\n4pqyNbxmLpf79u3bwsJCHG/GxsYIoYKCgt7e3traWv6DB76goKC+vr5r164Jvl+VTqdv3rz54sWL\nqampnZ2dPB6vsbHx9evX4tp9IPcEb1YKc382JSUFvzx9xowZ6enp/PIdO3ZMmjQJ12liYlJQUECS\nZF5eHpPJPHDgwPB6iouL58yZo6SkhBCaMmVKbGzssWPH8LjSmTNn1tfXnzhxAr9CcPr06TU1NSRJ\nBgQETJgwwdDQkEajaWhorFu3rr6+nl/hu3fvli9fTqfTTUxM/v73v//zn/9ECJmbm+MHBqWlpdOn\nT1dTU3NwcPj3v/894tudsCtXruAKIyIidHR0tLS0fHx88EsdzczM+I8fSJKcP39+ZGTkkP3q6+uL\niIgwNjam0Wi6urpeXl6VlZWHDx/GU8imTZsmzCwSeB6gGIb/jvI0/00WpmwJWr169bNnzyRRM8Sb\nYpD7+W9Sn7LFPxd9/PgxPpZKtz9AvshZvEldREREbW1tTU3N5s2b9+/fL+3ujAuQj0oKZGTKFoPB\n+OKLL1asWBEdHW1paSmtbow3CpOPSp6u38YPCq7fenp6WCyWdKsS+f0lJEkePHjQwsKCw+GQJMnl\nctesWcNfhfMHxMbGfrbmsrIyT0/Pc+fOzZs3z8bGZvgGISEhLBaLy+UKFsL7S8CoiXGCEvVznSAf\nFZACkiQTEhLwAG5tbe1169bxx2qOaoKS3KWngnxUQAqio6MjIyN37drV3Nx8586dly9fOjo64swS\no8opJXfpqSAfFaAah8NJSEjw9PTctGmTpqamlZXV8ePHW1tbRR4yLi/pqSAfFZCCysrKDx8+LFq0\niF9ia2uroqIyfOiZCGQ5PRXkowJSgO9TT5w4UbBQS0tLXLMQZTY9FeSjAlKAJ2EMiS5xTVCS5fRU\nkI8KSMHcuXMnTpz44MEDfklJSUl/f//ChQvx4lgmKMlyeirIRwWkgE6nh4eHX7ly5dy5c52dnRUV\nFUFBQQYGBgEBAXiD0U5Qkpf0VJCPCkjH3r174+LiYmJiJk+evGzZshkzZvDn76HR55SSo/RUkI8K\niBP183GkMtcJ8lGB8Uvqc50+AfJRAUCdcZ2PCigYGZnr9DGKlI8KngcAFBcXFxcXJ+1eiMLZ2dnZ\n2VkSNbu7uw/PhzF2cHwDgDoQbwBQB+INAOpAvAFAnRHul1y6dIn6fgBBeAyRwv8Q9+7dQwq9m42N\njUPHfAs+/FbgTCUASMWQ8SUEKaYXMwAqEQSRkZGhwJllFBVcvwFAHYg3AKgD8QYAdSDeAKAOxBsA\n1IF4A4A6EG8AUAfiDQDqQLwBQB2INwCoA/EGAHUg3gCgDsQbANSBeAOAOhBvAFAH4g0A6kC8AUAd\niDcAqAPxBgB1IN4AoA7EGwDUgXgDgDoQbwBQB+INAOpAvAFAHYg3AKgD8QYAdSDeAKAOxBsA1IF4\nA4A6EG8AUAfiDQDqQLwBQB3IbyofAgICqqur+YulpaUmJiba2tp4UVlZ+fvvvx+aKhrIHpq0OwCE\noq+vf+LECcGSx48f8/9tamoKwSYX4HxSPmzcuPFjq1RUVPz9/SnsCxAdnE/Kjblz51ZVVY34e1VX\nV1tYWFDfJTBacHyTG1999ZWysvKQQoIgbGxsINjkBcSb3NiwYQOPxxtSqKys/Je//EUq/QEigPNJ\neWJvb19SUjI4OMgvIQji5cuXhoaGUuwVEB4c3+TJn//8Z4Ig+ItKSkoODg4QbHIE4k2e+Pj4CC4S\nBPHVV19JqzNABBBv8mTy5MlOTk78uyYEQXh4eEi3S2BUIN7kzKZNm/Alt7KysouLy6RJk6TdIzAK\nEG9yxtPTU0VFBSFEkuSmTZuk3R0wOhBvckZdXX3NmjUIIRUVlbVr10q7O2B0IN7kj5+fH0LIw8ND\nXV1d2n0Bo0RKgLT3CQAxyMjIEHtoSGp+QFhYGIvFklDl8s7X13eM38+5c+fYbDaNJtPTOxITExFC\n27dvl3ZHROHr6yuJaiUyvoQgiIyMjPXr14u9ZsUw9u+nt7eXTqeLsUuSgJ8WXr58WdodEYWE/obh\n+k0uyX6wgRFBvAFAHYg3AKgD8QYAdSDeAKAOxJvcyMvL09TUvHr1qrQ7IikFBQWRkZGDg4MeHh7G\nxsZ0Ot3Q0NDd3V3wzUifNTg4mJiYaG9vL1iYm5t7+PDh4bN1qQfxJjcUeyDB3r17k5OTo6KiBgcH\n7969e+HChba2tqKiIg6Hs3Tp0qamJmEqqa2tXbp06Y4dO3p6egTL3dzc6HS6k5NTe3u7ZLovLIg3\nueHq6trR0UHBmEkOhzPk+CBphw4dSk9Pv3TpEpPJRAixWCwHBwcGg2FiYhIbG9vR0XHmzJnPVlJe\nXr5z586goKB58+YNXxsaGmpjY7N69eqBgQGx9194EG9gqLS0tObmZsqaq6ur27Nnz759+/BDRRqN\nJnjObGpqihCqr6//bD02NjZZWVl+fn6qqqojbhAdHV1WVpaUlCSmjosC4k0+FBUVGRsbEwSRkpKC\nEEpNTVVXV2cwGDk5OatWrdLQ0DAyMrp48SLeODk5mU6n6+npBQYGGhgY0Ol0/OITvDYkJERFRWXK\nlCl48euvv1ZXVycIorW1FSEUFhYWHh5eX19PEIS5uTlC6MaNGxoaGrGxsRLateTkZJIk3dzcRlzL\n4XAQQhoaGmNvSFtbe9myZUlJSVI8M4d4kw8ODg6//PILfzE4OHj79u0cDofJZGZkZNTX15uamm7b\nto3L5SKEQkJC/P39e3p6QkNDGxoaSktLBwYGvvzyy5cvXyKEkpOTBYcpHTt2bN++ffzFpKSktWvX\nmpmZkSRZV1eHEMK3GQRfUiRe169fnzVrFoPBGHHtr7/+ihBycHAQS1vz589/9epVeXm5WGoTAcSb\nfLO3t9fQ0NDV1WWz2d3d3S9evOCvotFos2fPVlVVtbS0TE1N7erqOn36tAhNuLq6dnZ27tmzR3y9\n/j/d3d2///67mZnZ8FVv375NT08PDQ1lsVgfO/qN1syZMxFCFRUVYqlNBDI9wBwID0/6xse34RYt\nWsRgMJ4+fUptpz6vubmZJMkRD24sFqu7u3v9+vUHDhyYMGGCWJrDDb19+1YstYkA4m28UFVVbWlp\nkXYvhurt7UUIjXiHQ09PLy0tbc6cOWJsTk1Njd+oVMD55LjA5XLb29tlMIcODoARn0Tr6upqaWmJ\nt7n+/n5+o1IBx7dxobCwkCRJOzs7vEij0T525kkxPT09giA6OjqGr5LESBrckL6+vthrFhIc3xTW\n4ODg+/fvBwYGHj9+HBYWZmxszE9bZW5u3tbWlp2dzeVyW1panj9/LvhBHR2dpqamhoaGrq4uLpeb\nn58vuecBDAbD1NS0sbFxSHldXZ2+vv6QSdZsNltfX7+0tFTk5nBDVlZWItcwRhBv8iElJcXW1hYh\nFBER4e7unpqait9WYG1t/ezZs5MnT4aHhyOEVq5cWVtbiz/S29trZWWlpqbm6OhoYWFx+/Zt/mVS\ncHDw8uXLN2zYMGvWrP379+PzKxaLhR8YBAUF6enpWVparl69uq2tTdK75urqWllZiZ+z8Y34iKy/\nv7+5uTknJ2fEeoqLix0cHKZOnVpSUlJeXm5gYLBkyZI7d+4IbnP//n1DQ0Nra2sx9n90xP5GFPxN\nSeJdKwqDgu8nICBAR0dHok18lre3t7e392c3q62tpdFoZ8+e/eyWPB7P0dExLS1NtP60trbS6fQj\nR44Is7GEfiM4viksWRgOLwxzc/OYmJiYmJgPHz58YjMej5ednd3V1cVms0VrKDo6et68eSEhIaJ9\nXCykE28LFiwgCIIgCAMDg9DQ0I9tVlNTY2trO3HiRCUlpZUrV36iwqysLFNTU0KAioqKnp7eH//4\nx/j4+Pfv30tgJ4DYREZG+vj4sNnsEW+cYIWFhVlZWfn5+R8bifJpCQkJZWVleXl54nqUJyKxHzFJ\n4Y7Fzs7OBEG8fv1asHBgYGD58uVDtvzXv/7l5+cnTLtmZmaampokSeJbBbdv3/b398dRff/+/dHs\ngWQJ8/2MRWRkJH78PWPGjMuXL0uuoU8T8nyS7+bNmxEREZLoSXZ2dlxc3MDAgPAfkdBvJLV4O3Xq\nFELo5MmTgoU//fQTQqiyslKw0MXFJTc3V5h2+fEm6PLly0pKSnp6eu3t7cJUQgFJx5uMGG28yRQJ\n/UZSu37DeSdyc3MFC3/88cepU6dmZmbySzgcTnl5uYuLi8gNeXt7+/v7Nzc3Hz9+XPTuAiAOUos3\nTU1NFxeXgoIC/o1g/ATW29tbMN5u3brl4uKCz45EnhiCnzvl5+fjRR6P9+233xobG6upqVlbW2dk\nZKDPzXBBCP3888+LFy9mMBgaGhpWVladnZ0fqwqAj5Hm/Uk2m83hcAoKCvDiTz/9tGLFCh8fn4qK\nipqaGlyYl5fHf+gp8sQQPOH32bNneHHnzp3fffddYmLi69ev165du3HjxgcPHnx6hkt3d7ebm5u3\nt3dbW1ttba2FhQUeGTRiVWP9XoACE/sZKin0uW9XV5eamtrWrVvxYnh4OJfLHRwcnDp16oEDB3Dh\nggUL+vv7hWx3xOs3jCAILS0tkiQ5HA6DwWCz2bi8p6dHVVU1ODiYJMldu3YhhDgcDl517NgxhFBd\nXR1Jkk+ePEEIXbt2TbDOT1T1aUJ+P/IOrt+Gk+b4yYkTJ7q6uuI/4v7+fhqNhhNQeHl5ZWZm7tq1\nq6qqav78+WO/gdvd3U2SJJ4jXF1d3dPTM3fuXLxKTU1typQpI05UEZzhYmpqqqent2nTptDQUH9/\n/xkzZoyqquHu3bs3xp2SfXjw1KVLl6TdEVki9ggmR/N/A75UKykpuXLlyu3bt3Hhzz//jBCqq6uL\nj4//8ccfhW/3Y8c3POLO2dmZJMn//ve/w78EOzs7ctjx7eTJkwih3377DS8+efJkzZo1NBqNIAhf\nX9+enp5PVPVpkvw9gdgo1P1JzNXVlclk5ubm3rlzZ+nSpbjQwcHBwMAgMzPzwYMHy5cvH3srN27c\nQAitWrUKIaSrq4sQSkxMFPwWhDnazJkz5+rVq01NTRERERkZGUeOHBG5KgTnkzJvTH9wHyfleKPT\n6W5ubpmZmWpqakpK/9MZJSUlT0/PH374QU9Pb+wpzt68eZOYmGhkZLRlyxaE0LRp0+h0ellZ2agq\naWpqqqqqQgjp6uoePHhwwYIFVVVVolUFxjPpj59ks9nV1dU4JzWfj49PVVWVh4eHYKEwE0NIkvzw\n4cPg4CBJki0tLRkZGUuWLFFWVs7OzsbXb3Q6ffPmzRcvXkxNTe3s7OTxeI2Nja9fv/50J5uamgID\nA58+fdrf3//o0aPnz5/b2dmJVhUY1yR0LBb+fKm/v9/GxgZHCB+Px7OxseHxeIKFeXl5TCaTf+tS\nUG5urrW1NYPBUFFRwcdJfENy8eLFMTEx7969E9y4r68vIiLC2NiYRqPp6up6eXlVVlYeO3YMD8yb\nOXNmfX39iRMncHxOnz69pqamoaHB3t5eW1tbWVl56tSpu3btwoODRqxKvN+P/JL380lJ/EaQ31QK\nxsn3A/lNh5P++SQA4wfEGwDUgXgDsgLyUQFAEchHBeSVGBNKUZObCvJRATkmxoRSFOSmgnxUQPpI\nkkxISMAJN7S1tdetW8cfCT2qhFKyn5tqXOWjkv7z7nFImO/n22+/VVFROXv2bHt7++PHjxcsWDB5\n8uQ3b97gtX5+fvr6+vyN4+PjEUItLS140cvLCyeUwgICAtTV1auqqnp7eysrK21tbZlM5osXL0So\n6tq1a0wmMyYmRpjdFPJ5t6mpqaWl5cfWZmVlIYQyMzOFaRH7wx/+YGNjM+KqyMhIhNCjR48+W4mE\n/obh+CaLOBxOQkKCp6fnpk2bNDU1raysjh8/3traeuLECdEqlNncVJCPCkhfZWXlhw8fFi1axC+x\ntbVVUVHhnweOhUzlpoJ8VED68G3riRMnChZqaWl1dXWJpX7ZyU0F+aiA9OE8TEOiS1wJpWQqNxXk\nowLSN3fu3IkTJwq+eqikpKS/v3/hwoV4cSwJpWQqNxXkowLSR6fTw8PDr1y5cu7cuc7OzoqKiqCg\nIAMDg4CAALzBqBJKIRnOTQX5qIBM2Lt3b1xcXExMzOTJk5ctWzZjxozCwkJ1dXW8drQJpWQ5NxXk\noxorBM/fPoni70dauakgH9VwcHwbF2RhaPzHQD4qACg1fvJRQbwpuKioqNOnT3d0dJiYmAgmZpA1\nsbGxISEhBw8e/NgGTk5O58+f5w/1HJWcnJy+vr7CwkJtbe0x9FEM4HmAgouLi4uLi5N2L4Ti7Ozs\n7OwsiZrd3d3d3d0lUfNowfENAOpAvAFAHYg3AKgD8QYAdSR1vyQxMVFOX/RJjfHw/RQXF6P/fesr\nwCTyfmX4ioEC2LFjB4vFEm+dEok3AMCI4PoNAOpAvAFAHYg3AKgD8QYAdf4fdiNZs/TpkOAAAAAA\nSUVORK5CYII=\n","text/plain":["<IPython.core.display.Image object>"]},"metadata":{"tags":[]},"execution_count":143}]},{"cell_type":"markdown","metadata":{"id":"pYKh54SKoma8","colab_type":"text"},"source":["## Loss"]},{"cell_type":"code","metadata":{"id":"1aNcIvmiow7Q","colab_type":"code","colab":{}},"source":["from tensorflow.keras.losses import MeanSquaredError"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"gesdrs3Vo0nq","colab_type":"code","outputId":"1abbe82e-5bc5-46e7-969a-55cd6a351457","executionInfo":{"status":"ok","timestamp":1583941114386,"user_tz":420,"elapsed":354,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["mse = MeanSquaredError()\n","loss = mse([0., 0., 1., 1.], [1., 1., 1., 0.])\n","print('Loss: ', loss.numpy()) "],"execution_count":145,"outputs":[{"output_type":"stream","text":["Loss:  0.75\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"ptfC0g-dxckr","colab_type":"text"},"source":["## Metrics"]},{"cell_type":"code","metadata":{"id":"Lm3Am3Gqxeda","colab_type":"code","colab":{}},"source":["from tensorflow.keras.metrics import MeanAbsolutePercentageError"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"9WMcuAkKiH09","colab_type":"code","outputId":"a90163d3-0444-41e5-b0cd-0e68831d66ad","executionInfo":{"status":"ok","timestamp":1583941116377,"user_tz":420,"elapsed":749,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["metric = MeanAbsolutePercentageError()\n","metric.update_state([0.5, 0.5, 1., 1.], [0.5, 1., 1., 0.])\n","print('Final result: ', metric.result().numpy())"],"execution_count":147,"outputs":[{"output_type":"stream","text":["Final result:  50.0\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"mctyzTveoegu","colab_type":"text"},"source":["## Optimizer"]},{"cell_type":"markdown","metadata":{"id":"PZ6IZ5pYcl1a","colab_type":"text"},"source":["When we implemented linear regression with just NumPy, we used batch gradient descent to update our weights. But there are actually many different [gradient descent optimization algorithms](https://ruder.io/optimizing-gradient-descent/) to choose from and it depends on the situation. However, the [ADAM optimizer](https://ruder.io/optimizing-gradient-descent/index.html#gradientdescentoptimizationalgorithms/#adam) has become a standard algorithm for most cases. "]},{"cell_type":"code","metadata":{"id":"ViOE7L5RdRgX","colab_type":"code","colab":{}},"source":["from tensorflow.keras.optimizers import Adam"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"8EviGS0ldLvN","colab_type":"code","colab":{}},"source":["# Optimizer\n","optimizer = Adam(lr=LEARNING_RATE)"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"nXbZVkMAyIpL","colab_type":"text"},"source":["## Training"]},{"cell_type":"markdown","metadata":{"id":"fDhb7qGZrxGS","colab_type":"text"},"source":["Here are the full list of options for [optimizer](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/optimizers), [loss](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/losses) and [metrics](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/metrics)."]},{"cell_type":"code","metadata":{"id":"1xwV9WvBhVaF","colab_type":"code","colab":{}},"source":["# Compile\n","model.compile(optimizer=Adam(lr=LEARNING_RATE),\n","              loss=MeanSquaredError(),\n","              metrics=[MeanAbsolutePercentageError()])"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"7UBMYJJpr2T-","colab_type":"text"},"source":["When we implemented linear regression from scratch, we used batch gradient descent to update our weights. This means that we calculated the gradients using the entire training dataset. We also could've updated our weights using stochastic gradient descent (SGD) where we pass in one training example one at a time. The current standard is mini-batch gradient descent, which strikes a balance between batch and stochastic GD, where we update the weights using a mini-batch of n (`BATCH_SIZE`) samples."]},{"cell_type":"code","metadata":{"id":"BcaPyVixsCS0","colab_type":"code","colab":{}},"source":["BATCH_SIZE = 10"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"PeGmesCXcSqm","colab_type":"code","outputId":"888002b6-b063-4f4a-c94f-e2d887b20ac3","executionInfo":{"status":"ok","timestamp":1583941122453,"user_tz":420,"elapsed":1651,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":1000}},"source":["# Training\n","history = model.fit(x=X_train, \n","                    y=y_train,\n","                    validation_data=(X_val, y_val),\n","                    epochs=NUM_EPOCHS,\n","                    batch_size=BATCH_SIZE,\n","                    shuffle=False,\n","                    verbose=1)"],"execution_count":152,"outputs":[{"output_type":"stream","text":["Train on 35 samples, validate on 7 samples\n","Epoch 1/50\n","35/35 [==============================] - 0s 8ms/sample - loss: 2.2032 - mean_absolute_percentage_error: 144.8060 - val_loss: 0.7335 - val_mean_absolute_percentage_error: 227.3441\n","Epoch 2/50\n","35/35 [==============================] - 0s 450us/sample - loss: 1.1970 - mean_absolute_percentage_error: 108.1093 - val_loss: 0.3955 - val_mean_absolute_percentage_error: 328.6021\n","Epoch 3/50\n","35/35 [==============================] - 0s 401us/sample - loss: 0.5289 - mean_absolute_percentage_error: 74.9418 - val_loss: 0.1668 - val_mean_absolute_percentage_error: 206.9364\n","Epoch 4/50\n","35/35 [==============================] - 0s 403us/sample - loss: 0.1644 - mean_absolute_percentage_error: 43.4698 - val_loss: 0.0530 - val_mean_absolute_percentage_error: 718.5472\n","Epoch 5/50\n","35/35 [==============================] - 0s 434us/sample - loss: 0.0381 - mean_absolute_percentage_error: 26.2976 - val_loss: 0.0190 - val_mean_absolute_percentage_error: 983.8639\n","Epoch 6/50\n","35/35 [==============================] - 0s 377us/sample - loss: 0.0444 - mean_absolute_percentage_error: 29.6342 - val_loss: 0.0161 - val_mean_absolute_percentage_error: 962.9730\n","Epoch 7/50\n","35/35 [==============================] - 0s 396us/sample - loss: 0.0828 - mean_absolute_percentage_error: 36.9853 - val_loss: 0.0176 - val_mean_absolute_percentage_error: 846.5008\n","Epoch 8/50\n","35/35 [==============================] - 0s 393us/sample - loss: 0.0976 - mean_absolute_percentage_error: 40.1439 - val_loss: 0.0161 - val_mean_absolute_percentage_error: 814.3967\n","Epoch 9/50\n","35/35 [==============================] - 0s 398us/sample - loss: 0.0808 - mean_absolute_percentage_error: 37.4335 - val_loss: 0.0139 - val_mean_absolute_percentage_error: 860.1505\n","Epoch 10/50\n","35/35 [==============================] - 0s 536us/sample - loss: 0.0519 - mean_absolute_percentage_error: 31.4122 - val_loss: 0.0150 - val_mean_absolute_percentage_error: 873.4119\n","Epoch 11/50\n","35/35 [==============================] - 0s 394us/sample - loss: 0.0309 - mean_absolute_percentage_error: 27.1245 - val_loss: 0.0203 - val_mean_absolute_percentage_error: 802.0851\n","Epoch 12/50\n","35/35 [==============================] - 0s 389us/sample - loss: 0.0239 - mean_absolute_percentage_error: 25.0748 - val_loss: 0.0277 - val_mean_absolute_percentage_error: 699.3417\n","Epoch 13/50\n","35/35 [==============================] - 0s 418us/sample - loss: 0.0261 - mean_absolute_percentage_error: 25.5582 - val_loss: 0.0335 - val_mean_absolute_percentage_error: 646.6029\n","Epoch 14/50\n","35/35 [==============================] - 0s 416us/sample - loss: 0.0295 - mean_absolute_percentage_error: 26.0150 - val_loss: 0.0349 - val_mean_absolute_percentage_error: 665.0445\n","Epoch 15/50\n","35/35 [==============================] - 0s 416us/sample - loss: 0.0298 - mean_absolute_percentage_error: 25.7870 - val_loss: 0.0326 - val_mean_absolute_percentage_error: 707.7242\n","Epoch 16/50\n","35/35 [==============================] - 0s 398us/sample - loss: 0.0277 - mean_absolute_percentage_error: 25.4551 - val_loss: 0.0284 - val_mean_absolute_percentage_error: 727.8849\n","Epoch 17/50\n","35/35 [==============================] - 0s 409us/sample - loss: 0.0255 - mean_absolute_percentage_error: 25.5023 - val_loss: 0.0244 - val_mean_absolute_percentage_error: 725.5954\n","Epoch 18/50\n","35/35 [==============================] - 0s 372us/sample - loss: 0.0244 - mean_absolute_percentage_error: 25.5995 - val_loss: 0.0218 - val_mean_absolute_percentage_error: 727.2172\n","Epoch 19/50\n","35/35 [==============================] - 0s 399us/sample - loss: 0.0243 - mean_absolute_percentage_error: 25.7011 - val_loss: 0.0207 - val_mean_absolute_percentage_error: 742.0944\n","Epoch 20/50\n","35/35 [==============================] - 0s 395us/sample - loss: 0.0245 - mean_absolute_percentage_error: 25.6953 - val_loss: 0.0208 - val_mean_absolute_percentage_error: 754.7386\n","Epoch 21/50\n","35/35 [==============================] - 0s 387us/sample - loss: 0.0244 - mean_absolute_percentage_error: 25.5661 - val_loss: 0.0215 - val_mean_absolute_percentage_error: 751.1937\n","Epoch 22/50\n","35/35 [==============================] - 0s 432us/sample - loss: 0.0241 - mean_absolute_percentage_error: 25.4915 - val_loss: 0.0225 - val_mean_absolute_percentage_error: 737.0789\n","Epoch 23/50\n","35/35 [==============================] - 0s 418us/sample - loss: 0.0240 - mean_absolute_percentage_error: 25.4577 - val_loss: 0.0235 - val_mean_absolute_percentage_error: 727.1889\n","Epoch 24/50\n","35/35 [==============================] - 0s 481us/sample - loss: 0.0241 - mean_absolute_percentage_error: 25.4853 - val_loss: 0.0242 - val_mean_absolute_percentage_error: 727.0169\n","Epoch 25/50\n","35/35 [==============================] - 0s 436us/sample - loss: 0.0242 - mean_absolute_percentage_error: 25.4768 - val_loss: 0.0244 - val_mean_absolute_percentage_error: 730.5203\n","Epoch 26/50\n","35/35 [==============================] - 0s 422us/sample - loss: 0.0243 - mean_absolute_percentage_error: 25.4726 - val_loss: 0.0242 - val_mean_absolute_percentage_error: 731.6016\n","Epoch 27/50\n","35/35 [==============================] - 0s 433us/sample - loss: 0.0242 - mean_absolute_percentage_error: 25.4893 - val_loss: 0.0237 - val_mean_absolute_percentage_error: 731.0724\n","Epoch 28/50\n","35/35 [==============================] - 0s 380us/sample - loss: 0.0242 - mean_absolute_percentage_error: 25.5046 - val_loss: 0.0233 - val_mean_absolute_percentage_error: 732.3527\n","Epoch 29/50\n","35/35 [==============================] - 0s 415us/sample - loss: 0.0241 - mean_absolute_percentage_error: 25.4985 - val_loss: 0.0231 - val_mean_absolute_percentage_error: 735.3268\n","Epoch 30/50\n","35/35 [==============================] - 0s 365us/sample - loss: 0.0241 - mean_absolute_percentage_error: 25.4811 - val_loss: 0.0230 - val_mean_absolute_percentage_error: 736.9933\n","Epoch 31/50\n","35/35 [==============================] - 0s 423us/sample - loss: 0.0241 - mean_absolute_percentage_error: 25.4728 - val_loss: 0.0231 - val_mean_absolute_percentage_error: 736.0269\n","Epoch 32/50\n","35/35 [==============================] - 0s 410us/sample - loss: 0.0241 - mean_absolute_percentage_error: 25.4779 - val_loss: 0.0233 - val_mean_absolute_percentage_error: 734.1309\n","Epoch 33/50\n","35/35 [==============================] - 0s 370us/sample - loss: 0.0241 - mean_absolute_percentage_error: 25.4853 - val_loss: 0.0234 - val_mean_absolute_percentage_error: 733.2979\n","Epoch 34/50\n","35/35 [==============================] - 0s 419us/sample - loss: 0.0241 - mean_absolute_percentage_error: 25.4873 - val_loss: 0.0235 - val_mean_absolute_percentage_error: 733.5831\n","Epoch 35/50\n","35/35 [==============================] - 0s 420us/sample - loss: 0.0242 - mean_absolute_percentage_error: 25.4869 - val_loss: 0.0235 - val_mean_absolute_percentage_error: 733.9268\n","Epoch 36/50\n","35/35 [==============================] - 0s 387us/sample - loss: 0.0242 - mean_absolute_percentage_error: 25.4891 - val_loss: 0.0235 - val_mean_absolute_percentage_error: 733.9293\n","Epoch 37/50\n","35/35 [==============================] - 0s 595us/sample - loss: 0.0242 - mean_absolute_percentage_error: 25.4926 - val_loss: 0.0234 - val_mean_absolute_percentage_error: 734.0235\n","Epoch 38/50\n","35/35 [==============================] - 0s 408us/sample - loss: 0.0242 - mean_absolute_percentage_error: 25.4937 - val_loss: 0.0234 - val_mean_absolute_percentage_error: 734.4495\n","Epoch 39/50\n","35/35 [==============================] - 0s 402us/sample - loss: 0.0242 - mean_absolute_percentage_error: 25.4919 - val_loss: 0.0233 - val_mean_absolute_percentage_error: 734.8490\n","Epoch 40/50\n","35/35 [==============================] - 0s 367us/sample - loss: 0.0242 - mean_absolute_percentage_error: 25.4904 - val_loss: 0.0234 - val_mean_absolute_percentage_error: 734.8605\n","Epoch 41/50\n","35/35 [==============================] - 0s 393us/sample - loss: 0.0242 - mean_absolute_percentage_error: 25.4911 - val_loss: 0.0234 - val_mean_absolute_percentage_error: 734.6075\n","Epoch 42/50\n","35/35 [==============================] - 0s 374us/sample - loss: 0.0242 - mean_absolute_percentage_error: 25.4928 - val_loss: 0.0234 - val_mean_absolute_percentage_error: 734.4434\n","Epoch 43/50\n","35/35 [==============================] - 0s 541us/sample - loss: 0.0242 - mean_absolute_percentage_error: 25.4940 - val_loss: 0.0234 - val_mean_absolute_percentage_error: 734.4872\n","Epoch 44/50\n","35/35 [==============================] - 0s 414us/sample - loss: 0.0242 - mean_absolute_percentage_error: 25.4945 - val_loss: 0.0234 - val_mean_absolute_percentage_error: 734.5970\n","Epoch 45/50\n","35/35 [==============================] - 0s 394us/sample - loss: 0.0242 - mean_absolute_percentage_error: 25.4951 - val_loss: 0.0234 - val_mean_absolute_percentage_error: 734.6555\n","Epoch 46/50\n","35/35 [==============================] - 0s 422us/sample - loss: 0.0242 - mean_absolute_percentage_error: 25.4960 - val_loss: 0.0234 - val_mean_absolute_percentage_error: 734.6952\n","Epoch 47/50\n","35/35 [==============================] - 0s 401us/sample - loss: 0.0242 - mean_absolute_percentage_error: 25.4966 - val_loss: 0.0234 - val_mean_absolute_percentage_error: 734.7706\n","Epoch 48/50\n","35/35 [==============================] - 0s 400us/sample - loss: 0.0242 - mean_absolute_percentage_error: 25.4968 - val_loss: 0.0234 - val_mean_absolute_percentage_error: 734.8504\n","Epoch 49/50\n","35/35 [==============================] - 0s 383us/sample - loss: 0.0242 - mean_absolute_percentage_error: 25.4969 - val_loss: 0.0234 - val_mean_absolute_percentage_error: 734.8776\n","Epoch 50/50\n","35/35 [==============================] - 0s 362us/sample - loss: 0.0242 - mean_absolute_percentage_error: 25.4974 - val_loss: 0.0234 - val_mean_absolute_percentage_error: 734.8608\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"Bk-sCSPxA5eS","colab_type":"code","outputId":"7b3da21e-6190-4dc1-beb8-50ec4b478195","executionInfo":{"status":"ok","timestamp":1583941122456,"user_tz":420,"elapsed":468,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":51}},"source":["# Training metrics\n","print (f\"metrics: {list(history.history.keys())}\")\n","print (f\"final val_loss: {history.history['val_loss'][-1]:.2f}\")"],"execution_count":153,"outputs":[{"output_type":"stream","text":["metrics: ['loss', 'mean_absolute_percentage_error', 'val_loss', 'val_mean_absolute_percentage_error']\n","final val_loss: 0.02\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"vj7cpeCsi6tf","colab_type":"text"},"source":["## Evaluation"]},{"cell_type":"code","metadata":{"id":"LeQm-H2vi6A7","colab_type":"code","colab":{}},"source":["# Predictions\n","pred_train = model.predict(X_train)\n","pred_test = model.predict(X_test)"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"dnW9b7ZQkA2l","colab_type":"code","outputId":"c49e63e8-71c0-425c-f9d3-ee03a81e16be","executionInfo":{"status":"ok","timestamp":1583941127580,"user_tz":420,"elapsed":230,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["# Train metrics\n","train_mse = tf.keras.metrics.MeanSquaredError()\n","train_mse.update_state(y_train, pred_train)\n","print(f'train_mse: {train_mse.result().numpy(): .2f}')"],"execution_count":155,"outputs":[{"output_type":"stream","text":["train_mse:  0.02\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"TGV3-zwXlJEU","colab_type":"code","outputId":"01b67a11-6af8-4a62-860a-b74c9583ca61","executionInfo":{"status":"ok","timestamp":1583941128292,"user_tz":420,"elapsed":411,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["# Test metrics\n","test_mse = tf.keras.metrics.MeanSquaredError()\n","test_mse.update_state(y_test, pred_test)\n","print(f'test_mse: {test_mse.result().numpy(): .2f}')"],"execution_count":156,"outputs":[{"output_type":"stream","text":["test_mse:  0.01\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"HdNCjH4ZjnRu","colab_type":"text"},"source":["Since we only have one feature, it's easy to visually inspect the model."]},{"cell_type":"code","metadata":{"id":"h0a-iUi4i6Iy","colab_type":"code","outputId":"6ea9c1c4-e59c-478f-a326-50a1282a3e71","executionInfo":{"status":"ok","timestamp":1583941129797,"user_tz":420,"elapsed":818,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":336}},"source":["# Figure size\n","plt.figure(figsize=(15,5))\n","\n","# Plot train data\n","plt.subplot(1, 2, 1)\n","plt.title(\"Train\")\n","plt.scatter(X_train, y_train, label='y_train')\n","plt.plot(X_train, pred_train, color='red', linewidth=1, linestyle='-', label='model')\n","plt.legend(loc='lower right')\n","\n","# Plot test data\n","plt.subplot(1, 2, 2)\n","plt.title(\"Test\")\n","plt.scatter(X_test, y_test, label='y_test')\n","plt.plot(X_test, pred_test, color='red', linewidth=1, linestyle='-', label='model')\n","plt.legend(loc='lower right')\n","\n","# Show plots\n","plt.show()"],"execution_count":157,"outputs":[{"output_type":"display_data","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAA3IAAAE/CAYAAAADjvF6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOzdeXxU9dn38e9FjBLXqCBKUKGW4gYS\njbhgva1b3Im0WnfQUqx3ebRPNRW0D7WtCoq9rVut1LrdblSLEUGNCloXFFmCgiKKCMqggkBQJEqW\n3/PHmWRmkkwyme3M8nm/XnlxfmfOzLkmCVxc89vMOScAAAAAQPbo5ncAAAAAAICuoZADAAAAgCxD\nIQcAAAAAWYZCDgAAAACyDIUcAAAAAGQZCjkAAAAAyDIUckCGM7MCM9tkZnv5HQsAAAAyA4UckGTB\noqv5q8nM6sLa53f19Zxzjc657Z1zn6YiXgAA0i3ZuTLsdd8yswuSGSuQqbbyOwAg1zjntm8+NrMV\nkkY5516Kdr2ZbeWca0hHbAAAZIKu5koAbdEjB6SZmV1vZlPM7DEz+0bSBWZ2RPBTxFoz+9zMbjez\nwuD1W5mZM7O+wfbDwcefM7NvzOxNM+vn41sCACCpgtMK/p+ZLTezr8zsETMrDj62nZk9bmbrg3lz\njpntbGZ/kXSopHuDPXt/8fddAKlFIQf440xJj0raSdIUSQ2SrpDUQ9JQSSdJurSD558n6f9J2kXS\np5L+nMpgAQBIs6sknSjpKEl9JNVLujX42Ch5o8pK5OXNMZK2OOeulDRXXu/e9sE2kLMo5AB/vO6c\ne8Y51+Scq3POzXXOzXHONTjnlkuaLOm/Onj+k865ec65ekmPSBqclqgBAEiPX0ka65xb7Zz7TtIf\nJf3czExeUddT0j7BvDnXOfetn8ECfmCOHOCPz8IbZravpL9IOkTStvL+bs7p4PlfhB1vlrR9tAsB\nAMgmwWJtT0nPmpkLe6ibpF0l/VPS7pKeNLPtJT0k6f855xrTHizgI3rkAH+4Vu17JC2W9EPn3I6S\nxkuytEcFAIDPnHNOUkDSsc654rCv7s65r5xz3zvnxjvn9pV0tKSzJJ3T/HS/4gbSjUIOyAw7SNoo\n6Vsz208dz48DACDX/V3SRDPbU5LMbDczOz14fLyZ7W9m3SR9LW+eeVPweV9K+oEfAQPpRiEHZIYr\nJY2Q9I283rkp/oYDAICvbpb0kqRZwRWeZ0s6OPhYiaSn5eXMxZKeVShv3irpIjPbYGY3pzdkIL3M\n670GAAAAAGQLeuQAAAAAIMtQyAEAAABAlqGQAwAAAIAsQyEHAAAAAFmGQg4AAAAAssxWfgfQkR49\neri+ffv6HQYAIMXmz5//lXOup99xZAvyIwDkj2g5MqMLub59+2revHl+hwEASDEzW+l3DNmE/AgA\n+SNajmRoJQAAAABkGQo5AAAAAMgyFHIAAAAAkGUo5AAAAAAgy1DIAQAAAECWoZADAAAAgCxDIQcA\nAAAAWYZCDgAAAACyTEZvCA4AyExVNQFNql6q1bV16l1cpMryAaooLfE7LAAA8gY9cgCALqmqCWjc\n1EUK1NbJSQrU1mnc1EWqqgn4HRoAAP5raJA++yzlt6GQAwB0yaTqpaqrb4w4V1ffqEnVS32KCACA\nDPHJJ9LRR0s33pjyWyWlkDOz+8xsjZktjvL4MWa20cwWBr/GJ+O+AID0W11bp20atuifT/5RfdcH\nIs4DAJC3HntMOuww6Wc/k+66K+W3S9YcuQck3SnpoQ6uec05d1qS7gcA8MmIFbN13RTvk8Zx5WNa\nzvcuLvIrJAAA/PPNN9KYMdKcOVJ1tVRampbbJqWQc869amZ9k/FaAIAM9e230k476brGRr1TMkAV\n50+SM29gR1FhgSrLB/gcIAAAafb229J550nHHivNny9tt13abp3OOXJHmNk7ZvacmR2QxvsCABL1\nz39K228vNTZKs2frk2dmqvfO28kklRQXacLwgaxaCQDIH42N0oQJ0umnSzfdJE2enNYiTkrf9gML\nJO3tnNtkZqdIqpLUv70LzWy0pNGStNdee6UpPADIT51uI7Bxo1Rc7B2fcII3ZMRMFRKFGwAgP61a\nJV10kdTUJM2bJ+25py9hpKVHzjn3tXNuU/D4WUmFZtYjyrWTnXNlzrmynj17piM8AMhLnW4jsM8+\noSJuwQLphRckM9/iBQDAd089JR1yiHTccdLMmb4VcVKaCjkz293My/5mNiR433XpuDcAoH3RthF4\n7JGZXsG2fLl3sqkpbRO3c00Mqzqbmd1uZsvM7F0zOzjdMQIAYrB5s/SrX0lXXSU9/bR07bVSQYGv\nISVlaKWZPSbpGEk9zGyVpD9IKpQk59zfJf1M0mVm1iCpTtI5zjmXjHsDAOLT3nYBi289S9tvCZ6f\nOlU688w0R5VzHlDHqzqfLG+qQX9Jh0m6O/gnACBTLFwonXuuVFYm1dRIO+7od0SSkrdq5bmdPH6n\nvEQGAMgQvYuLFAgWc/utWa7n7r889CCftSVFDKs6D5P0UPDDzbfMrNjM9nDOfZ6WAAEA0TU1Sbff\nLt1wg3TrrdIFF/gdUYR0LXYCAMgwleUDNG7qIi25/uSWc7847wadftUIVSiGhVCQDCWSPgtrrwqe\no5ADAD99+aU0cqS0YYO3P9wPfuB3RG2kc/sBAEAGqfjw9YgibuiEmV4RV1rS+UIoSDszG21m88xs\n3tq1a/0OBwBy13PPeXPDy8qk117LyCJOokcOAPJT+OqTTzwh/exneiPs4WgLoUyqXkqvXHIFJIUv\nedYneK4N59xkSZMlqaysjLGvAJBs330njR3rzRF/7DHpv/7L74g6RI8cAOSTu++OLOKck372szaX\ntbcQSkfnEbdpki4Krl55uKSNzI8DAB+8/750+OHeHnELF2Z8ESfRIwcA+SO8gHvxRen446NeGr4Q\nSuvziF0Mqzo/K+kUScskbZZ0sT+RAkCeck6aPFn6/e+lCROkX/wia/ZMpZADgFx33nneEJFmMaxI\n2bwQSvjwyqLCAlWWD0hFhDkrhlWdnaRfpykcAEC4deukUaOklSu9uXD77ut3RF1CIQcAOaDdFSYH\n95a6hY2gf+016aijYnq95nlwrFoJAMhJs2ZJI0ZIP/+59Pjj0jbb+B1Rl1HIAUCWa15hsrn3LFBb\npz7DTpI+Wxy6KI594SpKSyjcAAC5pb5eGj9eeugh6f77pRNP9DuiuFHIAUCWiLavW/gKk92aGrV8\n0rDQk+bMkYYM8SliAAAyyLJl3nSD3XaTamq8P7MYhRwAZIH2et3GTV0kKbSS5Pzbz9OudV+3PKff\n1dP1CUUcACDfOef1wF11lfSHP0i//nXWLGjSEQo5AMgCHe3r1ne7bnp5/Ckt548bdbc+3nVPlbDC\nJAAg39XWSpddJi1a5M2LGzjQ74iShkIOALJAtP3b3hh3XES779XTJbHCJAAAmj1bOv986dRTpblz\npaLc+oCTQg4AskDrfd12/G6T3r3tnJb2C9Pe0B/f+07GCpMAgHzX0CDdeKP0t795e8SdcYbfEaUE\nhRwAZIHwfd1W3HRa5IPO6URJJ57uS2gAAGSOlSulCy7wthNYsEDq3dvviFKmW+eXAAD8VlFaotuG\n7BhRxD078524thUAACAn/etf0qGHej1wL7zgSxFXVRPQ0Imz1G/sDA2dOEtVNYGU3YseOQDIBmaK\n2OnGOZ0S7VoAAPLJpk3S5ZdLr78uPfusVFbmSxgdrTCdiukO9MgBQCabPz9yieSNG+mFAwCg2dy5\nUmmplysXLPCtiJM6XmE6FeiRA4BM1XqPGwo4AAA8TU3SLbd4X3feKZ19tt8RRV1hOtr5RNEjBwCZ\n5vnnI4u477+niAMAoFkgIJ1wgvTMM16PXAYUcZK3wnRXzieKQg4AOpDOScuSvALu5JNDbeekrbdO\n7T0BAMgWTz8tHXKIdMwx0ssvS3vv7XdELSrLB6iosCDiXCr3dWVoJQBEkdZJyw8+KI0cGWo3Nkrd\n+KwNAABJ0ubN0lVXSc89J02dKh15pN8RtdH8f4NJ1Uu1Og37ulLIAUAUHU1aTuo/yimeC1dVE0hb\nUgEAIOnefVc691xp0CBp4UJpp538jiiqitKStOVYPu4FgChSPmn5hhsiizjnUlLEjZu6SIHaOjmF\nehVTPkQUAIBEOSfdfrt03HHS1VdLjz6a0UVcutEjBwBR9C4uUqCdoi0pk5bDC7i99pJWrkz8NduR\ntl5FAACSac0a6eKLpa++kt56S9pnH78jyjj0yAFAFCmZtDx6dNteuBQVcVL6l0IGACBh1dXS4MHe\n1+uvU8RFQY8cAESR9EnL4QXc8cdLL76YhCg7ltJeRQAAkun776VrrpGeeMIbRnnMMX5HlNEo5ACg\nA0mZtHz88dLMmaF2GveEqywfELHyppTapZABAIjLBx94C5r06yfV1Ei77up3RBmPoZUAkEpmoSLu\n0kvTvrF3RWmJJgwfqJLiIpmkkuIiTRg+kPlxAIDM4Jz0j39IP/6xdNll0r//TREXo6T0yJnZfZJO\nk7TGOXdgO4+bpNsknSJps6SRzrkFybg3AMQi7Uvwp3hLga5I51LIAADEbP166Ze/lD7+WHr1VWm/\n/fyOKKskq0fuAUkndfD4yZL6B79GS7o7SfcFgE6lfQn+8CLuz3/2tYgDACAjvfKKt5jJ3ntLc+ZQ\nxMUhKT1yzrlXzaxvB5cMk/SQc85JesvMis1sD+fc58m4PwB0JFc29gYAIOvV10vXXSfdf790333S\nSR31BaEj6ZojVyLps7D2quC5NsxstJnNM7N5a9euTUtwAHJbypfgb2qKLOJuu40iDgCA1j7+WDrq\nKG8xk5oairgEZdxiJ865yc65MudcWc+ePf0OB0AOiLbUftI29i4I22vOOenyyxN/XQAAcsn//q90\n+OHS+edLM2ZIvXr5HVHWS1chF5C0Z1i7T/AcAKRcSjb2/u67yF64J5+kFw4AgNY2bpQuuECaMEF6\n6SXvw87WUxEQl3TtIzdN0hgze1zSYZI2Mj8OQLqkdGNviQIOAID2vPmm1wNXXi7Nmydtu63fEeWU\nZG0/8JikYyT1MLNVkv4gqVCSnHN/l/SsvK0HlsnbfuDiZNwXAGKVlCX416+P3Nvm5ZelY45J7DUB\nAMg1jY1eD9wdd0j33CNVVPgdUU5K1qqV53byuJP062TcCwB8QS8cAACd+/RTbyjlVltJCxZIJexj\nmioZt9gJAGSUTz6JLOLee48iDgCA9jz5pFRWJp16qvTiixRxKZauOXIAkH3ohQMAoHPffitdcYX0\nn/94K1IeeqjfEeUFeuQAoLX58yOLuNWru1zEVdUENHTiLPUbO0NDJ85SVQ0L9QIActCCBdLBB3vz\n4hYsoIhLI3rkACBcEnrhqmoCGjd1kerqGyVJgdo6jZu6SJISX3AFAIBM0NQk/c//SDffLN1+u3TO\nOX5HlHfokQMAyRsKEl7Eff113EMpJ1UvbSnimtXVN2pS9dJEIgQAIDN8/rl00knSU09Jb79NEecT\nCjkAMJNOOy3Udk7aYYe4X251bV2XzgMAkDWmT/eGUh55pDcnrm9fvyPKWxRyAPLXPfdE9ML9sPJp\nDZ0wM+H5bL2Li7p0HgCAjFdXJ/2f/yONGSM98YR03XXeFgPwDYUcgPxkJv3qVy3NvldPV0O3gpb5\nbIkUc5XlA1RUWBBxrqiwQJXlA+J+TQAAfLN4sTRkiLRmjbRwoXTUUX5HBFHIAcg348ZF9MINvfEl\n9b16esQlic5nqygt0YThA1VSXCSTVFJcpAnDB7LQCQAguzgn3XWX9JOfSFdeKT3+uFRc7HdUCKI/\nFED+aGdFytVjZ7R7aaLz2SpKSyjcAADZa+1a6ZJLpC++kGbPlvr39zsitEKPHIDcd9ZZkUWccy0r\nUjKfDQCAVl58URo8WDrgAOmNNyjiMhSFHIDcZiY9+WSo3WpLAeazIdXM7CQzW2pmy8xsbDuPjzSz\ntWa2MPg1yo84AUBbtkiVldLFF0sPPSRNnChtvbXfUSEKhlYCyE0HHii9916oHWVPuObhj5Oql2p1\nbZ16FxepsnxATMMiq2oCcT0P+cPMCiTdJekESaskzTWzac6591tdOsU5NybtAQJAsw8/lM49V+rT\nx1vQpEcPvyNCJyjkAOSe8GGU++0nvd/6/8yR4pnPVlUT0Lipi1o2/m5e7bL59YCgIZKWOeeWS5KZ\nPS5pmKSOfykBIF2ck+6/X7r6aulPf/JWdG49pxwZiUIOQMaJu6erncVMUmVS9dKWIq5Z82qXFHII\nUyLps7D2KkmHtXPdT83saEkfSvq/zrnP2rkGAJJrwwbp0kulDz6QXnnFmxOHrMEcOQAZpbmnK1Bb\nJyfFvq9beBE3fHhKizgp+qqWia52ibz0jKS+zrlBkl6U9GB7F5nZaDObZ2bz1q5dm9YAAeSg117z\nFjTZYw/p7bcp4rIQhRyAjNJRT1e7zNquSPnvf6cwQg+rXSJGAUl7hrX7BM+1cM6tc859H2zeK+mQ\n9l7IOTfZOVfmnCvr2bNnSoIFkAcaGqTx46Wzz5b+9jfpttuk7t39jgpxoJADkFFi7ulyLrKA+93v\nUt4LF47VLhGjuZL6m1k/M9ta0jmSpoVfYGZ7hDXPkLQkjfEByCeffCIdfbQ0Z45UUyOdeqrfESEB\nzJEDkFF6Fxcp0E4xF9HTlca5cNEkstol8odzrsHMxkiqllQg6T7n3Htm9idJ85xz0yRdbmZnSGqQ\ntF7SSN8CBpC7Hn1U+s1vpHHjpCuukLrRn5PtKOQAZJTK8gERq0FKYT1dDQ1SYWHo4rvv9lbX8kk8\nq10i/zjnnpX0bKtz48OOx0kal+64AOSJr7+Wxozx5sFVV0ulpX5HhCShkAOQUaL2dB3cJ/JCH3rh\nAADIKnPmSOedJx1/vDR/vrTddn5HhCSikAOQcSJ6ujZtknbYIfTgtGnS6af7ExgAANmgsVG66SZv\nIZO//U366U/9jggpQCEHIHNlwFw4AACyymefSRde6OXQefOkPffs/DnISsxyBJB5vvgisoh7+22K\nOAAAOjN1qlRWJpWXSy+9RBGX4+iRA5BZEuiFq6oJsIokACD/rFwp9e0r9ewpPfOMdNhhfkeENKBH\nDkBmWLIksohbvrzLRdy4qYsUqK2TkxSordO4qYtUVRPo9LkAAGStK67wijhJWrqUIi6P0CMHwH9J\nmAs3qXppxJYFklRX36hJ1UvplQMA5J7WW/IwBSHvJKVHzsxOMrOlZrbMzMa28/hIM1trZguDX6OS\ncV8AqVdVE9DQibPUb+wMDZ04K7k9XK+9FlnEffVV3IlodTubiHd0HgCArPXCC6Ei7t57KeLyVMI9\ncmZWIOkuSSdIWiVprplNc8693+rSKc65MYneD0D6NA9XbO7pah6uKCnxXq4kr0jZu7hIgXaKtt7F\nRQm9LgAAGeWHP5Q+/tg7/uYbafvtmSOep5LRIzdE0jLn3HLn3BZJj0saloTXBeCzjoYrxu2++yKL\nuLq6pHySWFk+QEWFBRHnigoLVFk+IOHXBgDAd19+6eXPjz/2VqV0rqWIY454fkpGIVci6bOw9qrg\nudZ+ambvmtmTZsZaqEAWSPpwRTPpF78ItZ2TuneP77VaqSgt0YThA1VSXCSTVFJcpAnDB/KJJAAg\n+/3pT9Luu3vH8+dLzz/f8lBKPnRFVkjXYifPSHrMOfe9mV0q6UFJx7Z3oZmNljRakvbaa680hQeg\nPUkbrvj730s33BBqNzZK3ZK/aG5FaQmFGwAgdzQ1SQUFke1WUxOYI56/kvE/qYCk8B62PsFzLZxz\n65xz3web90o6JNqLOecmO+fKnHNlPXv2TEJ4AOKVlOGKZpFFnHMpKeIAAMgps2eHirhbbvHyZ+v5\n5Yr+4SpzxHNfMv43NVdSfzPrZ2ZbSzpH0rTwC8xsj7DmGZKWJOG+AFIsoeGKZ58dmXCcY1UtAABi\nccQR0tCh3vG6ddKVV0a9lDni+SvhoZXOuQYzGyOpWlKBpPucc++Z2Z8kzXPOTZN0uZmdIalB0npJ\nIxO9L4D0iGu4YpJXpAQAIC/U1ko77+wdH3ywNx+uE805mlUr809S5sg5556V9Gyrc+PDjsdJGpeM\newHIYPvvLy0J63CngAMAIDZ33CFdfrl3/J//SEcfHfNTmSOen9K12AmAXEcvHAAAXdd67niKFgRD\n7uG3BEBizJgLBwBAPN55J1S0XXstC4KhS+iRAxC/8AJun32kZcv8iwUAgGwybJg0Lbg+4OrV0h57\ndHw90AqFHICuYxglAADx+fZbafvtveOSEmnVKn/jQdai7xZA14QXcWecQREHAECsHnooVMRNn04R\nh4TQIwcgNvTCAQAQv/A8umWLVFjoXyzICfTIAehYU1Nk8rn66oSLuKqagIZOnKV+Y2do6MRZqqoJ\nJBgkAAAZ6qOPQnn0ssu8HEoRhySgRw5AdCnohauqCWjc1EWqq2+UJAVq6zRu6iJJYg8cAEBuufBC\n6eGHveOPP5Z+8AN/40FOoZADckhVTUCTqpdqdW2dehcXqbJ8QEzFUevnXX3M3jrjiB+GLvj736VL\nL01KjJOql7YUcc3q6hs1qXophRwAIDds3ixtt513bOaNbgGSjEIOyBHx9nS1ft4b446LvCDJc+FW\n19Z16TwAAFnlmmukCRO84zFjpDvu8Dce5CzmyAE5oqOerliet1PdN1px02kt568acWNKFjTpXVzU\npfMAAGQNs1ARt3EjRRxSikIOyBHx9nStrq3TiptO0zu3n9tyru/V0/Xv3QclNb5mleUDVFRYEHGu\nqLBAleUDUnI/AABSbuHCyHnlzkk77uhfPMgLDK0EckTv4iIF2inaOuzpWrlSn4T1wp1wyV36qOfe\nnT8vAc3DPOOZywcAQMbp1Utas8Y7fv55qbzc33iQNyjkgBxRWT4gYq6b1ElPV6sVKftePT225yVB\nRWkJhRsAILvV10tbbx1qs78q0oyhlUCOqCgt0YThA1VSXCSTVFJcpAnDB7YtmGpqIou4QEBVC1Z1\n/jwAAOD5619DRdxPf0oRB1/QIwfkkE57uqLsC1fRmz3cAACISXgu/eILb2gl4AN65ACfVNUENHTi\nLPUbO0NDJ85SVU0gdTd77rnIxLNxI58eAgDQFR9/3HZBE4o4+IgeOcAH8e75FpcovXAAACBGhx4q\nzZvnHT/6qHTuuR1fD6QBPXKAD+Ld861L7r03sojbsoUiDgCArmhq8nJpcxHX2EgRh4xBjxzgg3j3\nfIsZvXAAACTm4YelCy/0jg8/XHrzzS49vaomwFY7SCkKOcAHce35FosxY6S77gq1mz9JBAAAsQvP\nncuXS/36denpaZ1CgbzF0ErAB5XlA1RUWBBxLuG928wiizjnKOIAAOiKL75ou6BJF4s4KU1TKJD3\nKOQAH8S851ssjj22bdJhKCUAAF1TUSHtsYd3fMcdCeXSlE+hAMTQSsA3He35FvO4eubCAQCQGOek\nbmF9G1u2SIWFCb1kyqZQAGHokQMyTPO4+kBtnZxC4+oj9pnr2ZNeOAAAEvXss6Eirk8fL5cmWMRJ\nKZpCAbRCjxyQYToaV19RWhJZwBUVSZs3pzlCAAByQHg+ffddaeDApL108ygaVq1EKlHIARkm2vj5\nN8YdJ40LO0EPHAAAXVdbK+28c6idonza0RQKIBkYWglkmPbGz6+46bRQY8gQijggi5jZSWa21MyW\nmdnYdh7fxsymBB+fY2Z90x8lkCcuuyxUxI0fTz5FVktKj5yZnSTpNkkFku51zk1s9fg2kh6SdIik\ndZJ+7pxbkYx7A7mmsnxAy94zEQWcRMIBsoyZFUi6S9IJklZJmmtm05xz74dd9gtJG5xzPzSzcyTd\nJOnn6Y8WyHHhQyk3b/amJwBZLOEeubAkdbKk/SWda2b7t7qsJUlJulVekgLQjorSEk0488CIIm7l\nsJ9TxAHZaYikZc655c65LZIelzSs1TXDJD0YPH5S0nFmbAIJJM3s2W0XCKOIQw5IRo9cS5KSJDNr\nTlLhnzYOk3Rd8PhJSXeamTnH/0yBNsxUEd52Tnv7FQuARJVI+iysvUrSYdGucc41mNlGSbtK+iot\nEQK5LLyAe+016aij/IsFSLJkzJFrL0m1ntkZkaQkNScpAM0aGiITzoQJ9MIBaGFmo81snpnNW7t2\nrd/hAJmtrq5tLxxFHHJMxi12QqJCXjKL3LfGOWlsmzURAGSfgKQ9w9p9gufavcbMtpK0k7z55BGc\nc5Odc2XOubKePXumKFwgB/zxj9K223rHv/wlH4oiZyVjaGVXktSqjpKU5CUqSZMlqaysjL95yG3f\nfittv32o/cgj0nnn+RcPgGSbK6m/mfWTlwvPkdT6L/k0SSMkvSnpZ5JmMfUAiFN4L9yGDVJxsX+x\nACmWjEKOJAXEo/VaBvyVAHJOcM7bGEnV8lZ2vs85956Z/UnSPOfcNEn/lPS/ZrZM0np5eRRAVyxe\nHLmhNzkVeSDhQo4kBXTRl19Ku+8eas+Z4+0NByAnOeeelfRsq3Pjw46/k3RWuuMCckbfvtLKld7x\nM89Ip53W4eVArkjKPnIkKSBG9MIBAJAcDQ2R88ubmtrmWSCHZdxiJ0BOWro0MrksW0YRBwBAvP72\nt1ARd+qpXk6liEOeSUqPHIAO0AsHAEDyhOfVQEDq3du/WAAf0SMHpMrrr0cmm7VrKeIAAIjXihVt\n94ajiEMeo5ADUsFM+vGPQ23npB49/IsHAIBsdtRRUr9+3vGDD/LBKCCGVgLJ9eST0llh6/ps3iwV\nFfkXDwAA2cw5qVtYv0NjY2QbyGMUcsgrVTUBTapeqtW1depdXKTK8gGqKC1JzoszFw4AgOSZMkU6\nJ7hj1eDBUk2Nv/EAGYZCDnmjqiagcVMXqa6+UZIUqK3TuKmLJCmxYu5//ke68spQu6FBKihIJFQA\nAPJb+IejH30k/fCH/sUCZCgKOeSNSdVLW4q4ZnX1jZpUvTT+Qq6TXriU9gACAJBr1q6Vdtst1GZ0\nCxAVg4yRN1bX1nXpfIfGjGm7clY7Rdy4qYsUqK2TU6gHsKom0PX7AQCQ6845J1TE3XILRRzQCXrk\nkDd6Fxcp0E7R1ru4i4uRxHwNjqMAACAASURBVDgXLiU9gAAA5KLw3Pr999LWW/sXC5Al6JFD3qgs\nH6Ciwsi5a0WFBaosHxDbC5x0Uqe9cOGS2gMIAEAuevHFUG7dZRcvr1LEATGhRw55o7kXLK45a+EF\n3A47SF9/3elTktYDCABALgrPrQsWSKWl/sUCZCEKOeSVitKSrg1r7NNHCoTNaevCeP3K8gERq2RK\nXewBBAAgi0Vd8Oubb6QddwxdyFw4IC4MrQSiMQsVcUcc0eVEU1FaognDB6qkuEgmqaS4SBOGD2R+\nHAAg50Vb8Ovj80aFirjf/Y4iDkgAPXJAa0nc2LvLPYAAAOSA9hb8WnL9yaHGpk3SdtulOSogt9Aj\nB4QLL+IuvphPCgEAiEP4wl6DPv9QK246LfSgcxRxQBLQIwdISe2FAwAg3zUv+BVewJ1z7o36bNBh\nesPHuIBcQo8c8ptzkUXcn/9MEQcAQILGHr1XRBHX9+rpemefUhb8ApKIHjnkL3rhAABIvp//XKf/\n61+SpC936qnDf3W/Srqy5Q+AmFDIIf98/73UvXuo/dBD0oUX+hcPAAC5IvxD0lWr1KukRJ/4Fw2Q\n0yjkkF/ohQMAIPneesvbqqcZ+RVIOebIIT+sXx9ZxD3zDEkGAIBkMAsVcX/5C/kVSBN65JD76IUD\nACD5GhqkwsJQu6mpbc4FkDL0yCF3LV8emVDmzaOIAwDktaqagIZOnKV+Y2do6MRZqqoJxPdCV10V\nWcS1XgUaQMrRI4fclMZeuKqagCZVL9Xq2jr1ZlUuAECGqqoJaNzURaqrb5QkBWrrNG7qIknqWt4K\nz7FLlkj77pvMMAHEiB455JY5cyITzIoVKS/ixk1dpEBtnZxCSTHuTzgBAEiRSdVLW4q4ZnX1jZpU\nvTS2F/jgg8gc6xxFHOAjCjnkDjPp8MNDbeekvfdO6S0TTooAAKTJ6tq6Lp2PYCbtt593fOWVTFUA\nMgCFHLJfVVXkJ4QbNqQtwSSUFAEASKPexUVdOi+p7dy3+nrplluSHBmAeCRUyJnZLmb2opl9FPxz\n5yjXNZrZwuDXtETuCUQwk848M9R2TiouTtvt40qKAAD4oLJ8gIoKCyLOFRUWqLJ8QPtPuPVWqVvY\nfxWdk7ZieQUgUyTaIzdW0kznXH9JM4Pt9tQ55wYHv85I8J6AdPvtkZ8Qfv+9L8M8upwUAQDwSUVp\niSYMH6iS4iKZpJLiIk0YPrD9hU7MpN/+1jt+6y2GUgIZKNGPVYZJOiZ4/KCkVyRdneBrAh3LoH3h\nmpMfq1YCALJBRWlJxzlq9WqpJOxxCjggYyVayPVyzn0ePP5CUq8o13U3s3mSGiRNdM5VJXhf5KPL\nL5fuuCPUzpCNRztNigAAZIO995Y+/dQ7PvtsacoUf+MB0KFOCzkze0nS7u08dG14wznnzCzaxzZ7\nO+cCZvYDSbPMbJFz7uMo9xstabQk7bXXXp2Fh3yRQb1wAADknPA8++230rbb+hcLgJh0OkfOOXe8\nc+7Adr6elvSlme0hScE/10R5jUDwz+Xyhl+WdnC/yc65MudcWc+ePeN4S8gpJ57Yds8aijgAAJLj\n0Ufb5lmKOCArJLrYyTRJI4LHIyQ93foCM9vZzLYJHveQNFTS+wneF/nATHrxxVCbAg4AgOQxk84/\n3zuePp08C2SZROfITZT0LzP7haSVks6WJDMrk/Qr59woSftJusfMmuQVjhOdcxRyiG733aUvv2xp\n9rt6ureISE2AuWgAACSqtlbaOWzHKAo4ICslVMg559ZJOq6d8/MkjQoez5Y0MJH7II+EDe+o71ag\n/pVeJ2+gtk7jpi6SJIo5AADidfXV0s03e8dDhkhz5vgbD4C4sasjMkOrxUyGTpipQG1dxLm6+kZN\nql5KIQcAQDzCc+3XX0s77OBfLAASlugcOSBx4YmlrExyTqtbFXHNop0HAABR1NS0XdCEIg7IehRy\n8I9Z28Qyd64kqXdxUbtPiXYeAAC0Y9ddpYMP9o5feIH5cEAOoZBD+jkXWcBddFGbxFJZPkBFhQUR\n54oKC1RZPiAdEQIAkN22bPFy7fr1Xts56YQT/I0JQFIxRw5JUVUT0KTqpVpdW+etMFk+oP25bDFu\n7N383JheEwAAhPzlL9JVV3nHZ58tTZnibzwAUoJCDgmrqglo3NRFqqtvlBRlhcmGBqmwMPSkP/9Z\n+v3vO3zditISCjcAWcvMdpE0RVJfSSskne2c29DOdY2SFgWbnzrnzkhXjMhB4R+Yrlkj9ezpXywA\nUoqhlUjYpOqlLUVcs+YVJiV5SSW8iHOu0yIOAHLAWEkznXP9Jc0MtttT55wbHPyiiEN8Pvqo7bxz\nijggp1HIIWHRVpLc+OW6yKQyeTKTrAHkk2GSHgwePyipwsdYkMsOPlj60Y+84ylTyLVAnmBoJRLW\nu7iozZ5vK246LfIikgqA/NPLOfd58PgLSb2iXNfdzOZJapA00TlXlZbokP2amqSCgsh267noAHIW\nPXJIWPgKk7t//VVkEff88xRxAHKWmb1kZovb+RoWfp1zzkmK9o/h3s65MknnSfqrme0T5V6jzWye\nmc1bu3Ztct8Iss9DD4WKuKFD264IDSDn0SOHhDUvSFJxcJ/IByjgAOQ459zx0R4zsy/NbA/n3Odm\ntoekNVFeIxD8c7mZvSKpVNLH7Vw3WdJkSSorK+Mf2HwWXrB98onUt69voQDwDz1ySNz770cWce+8\nQxEHANI0SSOCxyMkPd36AjPb2cy2CR73kDRU0vtpixDZZfXqtguaUMQBeYtCDokxkw44INR2Tho0\nyL94ACBzTJR0gpl9JOn4YFtmVmZm9wav2U/SPDN7R9LL8ubIUcihrdNOk0qCW/LcdRcfmAJgaCXi\nNGuWdNxxofbnn0u7797upTFvFg4AOcQ5t07Sce2cnydpVPB4tqSBaQ4N2cQ5qVvY5+719dJW/PcN\nAIUc4tF6MnUHnwrGtFk4AABoa/p06fTTveO995ZWrPA1HACZhaGViN3//m9kEbdpU6dDOzrdLBwA\nALRlFiriFi+miAPQBj1yOSrpwxm70AsXLtpm4dHOAwCQ1zZskHbZJdRmLhyAKOiRy2JVNQENnThL\n/cbO0NCJs1RVE2g5P27qIgVq6+QUGs7Y/HiX/OlPkUVcQ0OXkkrv4qIunQcAIG+NHh0q4q67jiIO\nQIfokctSHc0962g4Y5d65eLshQtXWT4gIk5JKiosUGX5gC6/FgAAOSs8527eLBXxgSeAjtEjl6U6\nKtYSHs54wQWRCaWpKe5PBStKSzRh+ECVFBfJJJUUF2nC8IEsdAIAgCS9/noo55p5+ZYiDkAM6JHL\nUh0Va72LixRo5/GYhjMmoReutYrSEgo3AABaC8+5s2dLRxzhXywAsg49clmqo7lnleUDVFRYEHG+\n0+GMpaWRCcU5xuYDAJAKmze3zbkUcQC6iEIuS3VUrHV5OKOZtHBhqE0BBwBAaowfL223nXd82WXk\nXABxY2hllmouyqJtMRDTcMZu3SITCMkEAIDUCe+Fq62VdtrJv1gAZD0KuSyW0Nyz8GRSUiKtWpWc\noAAAQKRFi6RBg0JtPjgFkAQMrcw3Zm3H5VPEAQCQGn36hIq4GTMo4gAkDT1y+SS8gLvoIunBB/2L\nBQCAXFZfL229dajd1NR2ZWgASEBCPXJmdpaZvWdmTWZW1sF1J5nZUjNbZmZjE7kn4tBeLxxFHAAA\nqXHHHaEirqLCy7sUcQCSLNEeucWShku6J9oFZlYg6S5JJ0haJWmumU1zzr2f4L3RGee8BU2a/fGP\n3mpZAAAgNcILts8/l3bf3b9YAOS0hAo559wSSbKOP2UaImmZc2558NrHJQ2TRCEXg6qaQNSVKTuU\ngo29AQBAFJ98Iv3gB6E2eRdAiqVjsZMSSZ+FtVcFz6ETVTUBjZu6SIHaOjlJgdo6jZu6SFU1gehP\n2rIlsoh74AGSCQAAqXTEEaEi7uGHybsA0qLTHjkze0lSe+MCrnXOPZ3sgMxstKTRkrTXXnsl++Wz\nyqTqpaqrb4w4V1ffqEnVS9vvlaMXDgCApIhpRExTk1RQEGo3NkZOaQCAFOq0kHPOHZ/gPQKS9gxr\n9wmei3a/yZImS1JZWVleVyKra+tiO//115Gbir78snTMMVFfN+7hmgAA5IHmETHNH6Y2j4iRFMqX\njz0mnXeed1xWJs2d60eoAPJYOrYfmCupv5n1k1fAnSPpvDTcN+v1Li5SoJ1irndxUajRxV64mJIT\nAAB5rNMRMeG5d9kyaZ990hwhACS+/cCZZrZK0hGSZphZdfB8bzN7VpKccw2SxkiqlrRE0r+cc+8l\nFnZ+qCwfoKLCgohzRYUFqiwfIH35ZWQiWbw4pqGUHSUnAAAQfUTMlsDqttv5UMQB8Emiq1Y+Jemp\nds6vlnRKWPtZSc8mcq981NxD1mYY5MF9Ii/swly4mIdrAgCQp9obEXP3Uzfq5A9ne42//lW64gof\nIgOAkHQMrUQCKkpLQkMeP/pI+lFYEffFF1KvXl16vZiGawIAkMcqyweEpiE4pxU3nx56cMsWqbDQ\nv+AAIIillbKFmfSjH4XaznW5iJM6Ga4JAABUUVqiCcMH6sw1i1uKuO927enlXoo4ABmCQi7TzZkT\nOR7/m28S2lagOTmVFBfJJJUUF2nC8IEsdAIAQJiKib/VrfeP9RoLF6r7V2v8DQgAWmFoZSZL0b5w\nEcM1AQBASF2dtO223nFxsbRhg7/xAEAU9MhlomeeiSzitmxhc28AAFLt8cdDRdxTT1HEAcho9Mhl\nmhT1wgEAgA5stZXUGNye5/vvpa239jceAOgEPXKZ4h//iCzimpoo4gAASLVPPvHyb2OjdPHFXu6l\niAOQBeiRywT0wgEAkH7//d/S3Xd7xx9+KPXv7288ANAFFHJ+uusuacyYUJsCDgCA1Gts9IZSNiP/\nAshCDK30i1moiDvlFJIIAADpUFMTKuIefJD8CyBrUcil2zXXRA6ldE6aMcO/eAAAyBe/+pV08MHe\n8XffSRdd5G88AJAACrl0MpMmTPCOL7uMTwEBAEiHDRu8HHzPPV4edk7aZhu/owKAhDBHrouqagKa\nVL1Uq2vr1Lu4SJXlAzrfXHv8eOnPfw61KeAAAEiPxx+Xzj3XO/7kE6lv3/hyOQBkGAq5LqiqCWjc\n1EWqq/f2mQnU1mnc1EWSFD0BhA+j/Oc/pUsuSXWYAACgqUk68EBpyRLpv/5LevllySy+XA4AGYih\nlV0wqXppyz/8zerqGzWpemnbi0eNajsXjiIOAIDUW7JEKijw/pwxQ3rllZac3KVcDgAZjB65Llhd\nWxfb+fACbto06fTTUxgVAABocfXV0s03e8ebNknbbRfxcMy5HAAyHD1yXdC7uKjj8yed1LYXjiIO\nAIDU27TJy8E33+ytEO1cmyJOiiGXA0CWoJDrgsryASoqLIg4V1RYoMoT+nvJo7raO/naayxoAgBA\nujzzjLTDDt7xBx9IN9wQ9dKoubx8QCojBICkY2hlFzRPgg5f6eq5yb/Sjtd/FLqIAg4AgPRwTjrq\nKGn2bGnQIGnhwsiRMe1oL5ezaiWAbEQh10UVpSXeP/b19dLWW4ceWLTIWx0rDiyDDABAFy1fLu2z\nj3f8r39JZ50V81NbcjkAZDGGVsZj++0jizjnEirixk1dpEBtnZxCyyBX1QSSEysAwBdmdpaZvWdm\nTWZW1sF1J5nZUjNbZmZj0xlj1rrhhlARt2FDl4o4AMgVFHJdUVfnDdn49luvvWJFwkMpWQYZAHLW\nYknDJb0a7QIzK5B0l6STJe0v6Vwz2z894WWh5jz8+99Lv/61l4OLi/2OCgB8QSEXq4EDpW239Y53\n2MFLHnvvnfDLsgwyAOQm59wS51xnn8oNkbTMObfcObdF0uOShqU+uiw0c2YoDy9cKN15p7/xAIDP\nKOQ6s3Gj9+nf4sVeu7ZW+vrrpL08yyADQF4rkfRZWHtV8BzCnXaadPzx0l57SQ0N0kEH+R0RAPiO\nQq4jO+8cGrKx//5eL9xOO8X1UlU1AQ2dOEv9xs7Q0ImzWubAsQwyAGQvM3vJzBa385X0XjUzG21m\n88xs3tq1a5P98pkpEPA+TJ0xQ7rvPmnlSqmgoPPnAUAeYNXK9mzcGDnmfvNmqSj+HrLmBU2a58I1\nL2gisQwyAGQz59zxCb5EQNKeYe0+wXPt3WuypMmSVFZWlvt73dx+u3TFFd7xmjVSz57+xgMAGYZC\nrrXTTvM++ZO8YRwvvpjwS3a0oEnzEsgUbgCQl+ZK6m9m/eQVcOdIOs/fkHxWX++NiPn2W+n886WH\nH/Y7IgDISBRyzdaskXr1CrUbGpI2fIMFTQAg/5jZmZLukNRT0gwzW+icKzez3pLudc6d4pxrMLMx\nkqolFUi6zzn3no9h++vNN6Ujj/SOZ8+WjjjC33gAIIMlVMiZ2VmSrpO0n6Qhzrl5Ua5bIekbSY2S\nGpxzUffTSZfwTbifebRSB362xHvgjjukMWOSeq/exUUKtFO0saAJkNvq6+u1atUqfffdd36HkjG6\nd++uPn36qLCw0O9QUs4595Skp9o5v1rSKWHtZyU9m8bQMtMFF0iPPOLt1bp+vZQHvyMAPORLT1dz\nZKI9cs175NwTw7U/cc59leD9kqJ5zlrxui/0yd0Xh87P/0wVB/dJ+v0qywdEzJGTWNAEyAerVq3S\nDjvsoL59+8rM/A7Hd845rVu3TqtWrVK/fv38DgeZYu1aabfdvOPbbpMuv9zfeACkHfkyvhyZ0KqV\nMe6Rk3EmVS/VOW9O1ZvBIu7/nvpb9b16uia98GFK7ldRWqIJwweqpLhIJqmkuEgThg9kXhyQ4777\n7jvtuuuueZuUWjMz7brrrnn/iSvC3H9/qIhbtYoiDshT5Mv4cmS65sg5SS+YmZN0T3DlLX+sW6c3\nxh0nSZo+4CiNqRjb8lDznLXwYZfJWkWSBU2A/JTPSak9fD8gSWpslPr1kz77TDr1VGn6dL8jAuAz\n8kPXvwed9sglaY+co5xzB0s6WdKvzezoDu6Xun1ybrhB6tFDknTcqLsjijjJm7PWPOwyUFsnp9BW\nAc37vgFAvurbt6+++qrjEfKxXIM898470lZbeUXczJkUcQBySjpzZaeFnHPueOfcge18PR3rTZxz\ngeCfa+RN/B7SwbWTnXNlzrmynsnaM+aLL7wNRX//e2nUKFUtWKXVu/eNuKR5zlpHWwUAAIAEjBkj\nDR7sHdfVScce6288AJDFEpojFwsz287Mdmg+lnSivEVS0uOvf5X22MM7Xr5c+sc/OpyzxlYBAHLJ\nihUrtO+++2rkyJH60Y9+pPPPP18vvfSShg4dqv79++vtt9/W+vXrVVFRoUGDBunwww/Xu+++K0la\nt26dTjzxRB1wwAEaNWqUnAvtQf3www9ryJAhGjx4sC699FI1NjZGCwGQamu9D1Tvuku6/nrJOal7\nd7+jAgBJ2ZsrEyrkzOxMM1sl6Qh5e+RUB8/3NrPmpZR7SXrdzN6R9LakGc655xO5b5esWSNdcYWX\nNMJWgKkoLdEbY4/VJxNP1Rtjj22ZvxZtSwC2CgCQrZYtW6Yrr7xSH3zwgT744AM9+uijev3113XL\nLbfoxhtv1B/+8AeVlpbq3Xff1Y033qiLLrpIkvTHP/5RRx11lN577z2deeaZ+vTTTyVJS5Ys0ZQp\nU/TGG29o4cKFKigo0COPPOLnW0Qme+IJb4NvSfr4Y+naa/2NBwDakY25MqHFTmLZI8c5t1zSQYnc\nJyE33tily9kqAEDKpGIid9gnf9H069dPAwcOlCQdcMABOu6442RmGjhwoFasWKGVK1fq3//+tyTp\n2GOP1bp16/T111/r1Vdf1dSpUyVJp556qnYO/md85syZmj9/vg499FBJUl1dnXZrXnkQaOacdNBB\n0qJF0tCh0muvpebvAIDc40O+zMZcma5VK7NGc89csletBIBYiq5U2GabbVqOu3Xr1tLu1q2bGhoa\nurw5t3NOI0aM0IQJE5IaJ3LI0qXSvvt6x9OmSaef7m88ALKLD/kyG3NlyufIZaNowy4BIBf9+Mc/\nbhnu8corr6hHjx7acccddfTRR+vRRx+VJD333HPasGGDJOm4447Tk08+qTVr1kiS1q9fr5UrV/oT\nPDLPNdeEirhvvqGIA5ATMjFX0iMHAHnuuuuu0yWXXKJBgwZp22231YMPPihJ+sMf/qBzzz1XBxxw\ngI488kjttddekqT9999f119/vU488UQ1NTWpsLBQd911l/bee28/3wb89u230vbbe8e/+510003+\nxgMASZSJudKcT0N9YlFWVubmzZvndxgAEJclS5Zov/328zuMjNPe98XM5jvnynwKKetkXH587jnp\nlFO84/fek/bf3994AGQV8mVIV3IkPXIAACA+zkk/+Yn0n/9I++0nLV4sdWPWBgCkA//aAgCArlux\nwiva/vMf6bHHpPffp4gDgDSiRw4AgDxUVROIf4Xmm26Sxo71jtevD+0TBwBIGwo5AADyTFVNIGLP\n1EBtncZNXSRJHRdz338vde/uHV96qfT3v6c6VABAFIyBAAAgz0yqXtpSxDWrq2/UpOql0Z/0n/+E\nirj58yniAMBn9MgBAJBnVtfWdem8Kiqkp5+W9thD+uwzqaAghdEBAGJBjxwAAHmmd3FRbOc//1wy\n84q4f/xDWr2aIg5A3lqxYkXL5t/xuPHGG5MYDYUcAOS9RBLTkUcemeRokA6V5QNUVBhZkBUVFqiy\nfEDoxN13S717e8dffCGNGpXGCAEg81DIAQDaVVUT0NCJs9Rv7AwNnThLVTWBtNy3o8TU0NDQ4XNn\nz56dipCQYhWlJZowfKBKiotkkkqKizRh+EBvoZOGBmmXXaT//m/p7LO9veJ69fI7ZABokex8OX78\neP31r39taV977bW67bbb2lw3duxYvfbaaxo8eLBuvfVWNTY2qrKyUoceeqgGDRqke+65R5L0+eef\n6+ijj9bgwYN14IEH6rXXXtPYsWNVV1enwYMH6/zzz08o3mbMkQOADBD3KoIdGD9+vHbZZRf95je/\nkeQlpt12201XXHFFxHVjx47VkiVLNHjwYI0YMUI777yzpk6dqk2bNqmxsVEzZszQsGHDtGHDBtXX\n1+v666/XsGHDJEnbb7+9Nm3apFdeeUXXXXedevToocWLF+uQQw7Rww8/LDOL91uCFKsoLWn7u/X2\n29Jhh3nHr78uDR2a/sAAoAOpyJeXXHKJhg8frt/85jdqamrS448/rrfffrvNdRMnTtQtt9yi6dOn\nS5ImT56snXbaSXPnztX333+voUOH6sQTT9TUqVNVXl6ua6+9Vo2Njdq8ebN+/OMf684779TChQvj\nfOdt5XQhl9AeOQCQRh2tIpjuxPTAAw9owYIFevfdd7XLLruooaFBTz31lHbccUd99dVXOvzww3XG\nGWe0KdJqamr03nvvqXfv3ho6dKjeeOMNHXXUUXHFDh+MHCk9+KC0zTbS119LW2/td0QA0EYq8mXf\nvn216667qqamRl9++aVKS0u16667dvq8F154Qe+++66efPJJSdLGjRv10Ucf6dBDD9Ull1yi+vp6\nVVRUaPDgwXHF1ZmcLeRSUa0DQKp0eRXBGMSbmCTphBNO0C677CJJcs7pmmuu0auvvqpu3bopEAjo\nyy+/1O677x7xnCFDhqhPnz6SpMGDB2vFihUUctlg3TqpRw/v+C9/kX77W3/jAYAOpCJfStKoUaP0\nwAMP6IsvvtAll1wS03Occ7rjjjtUXl7e5rFXX31VM2bM0MiRI/Xb3/5WF110UULxtSdn58jFtUcO\nAPgk5lUEu6g5Md1///0xJyZJ2m677VqOH3nkEa1du1bz58/XwoUL1atXL3333XdtnrPNNtu0HBcU\nFHQ6vw4Z4KGHQkXcp59SxAHIeKnKl2eeeaaef/55zZ07t93CTJJ22GEHffPNNy3t8vJy3X333aqv\nr5ckffjhh/r222+1cuVK9erVS7/85S81atQoLViwQJJUWFjYcm0y5Gwhl6pqHQBSIaZVBOMQT2Jq\nbePGjdptt91UWFiol19+WStXrkwoJmSQESOk8nJvQZM99/Q7GgDoVKry5dZbb62f/OQnOvvss1UQ\nZZuVQYMGqaCgQAcddJBuvfVWjRo1Svvvv78OPvhgHXjggbr00kvV0NCgV155RQcddJBKS0s1ZcqU\nlrnpo0eP1qBBg1jspDO9i4sUaKdoS7RaB4BUaB7ynex5vc2Jqbi4OKbENHLkSO28884Rj59//vk6\n/fTTNXDgQJWVlWnfffdNKCZkEOf8jgAAuiRV+bKpqUlvvfWWnnjiiajXFBYWatasWRHnbrzxxjbb\nCowYMUIjRoxo8/ybbrpJN910U0JxhsvZQq6yfEDEHDkpOdU6AKRKu6sIJijexDRy5MiW4x49eujN\nN99s97mbNm2SJB1zzDE65phjWs7feeed8QcNAEAHkp0v33//fZ122mk688wz1b9//6S9bqrlbCGX\nqmodALJFtiYmAADSaf/999fy5ctb2osWLdKFF14Ycc0222yjOXPmpDu0DuVsISel5tNtAMgW2ZqY\nAADw08CBA5O631uq5HQhBwAIyZbEBAAAOpezq1YCQCZwLCYRge8HAKA95Ieufw8o5AAgRbp37651\n69aRnIKcc1q3bp26d+/udygAgAxCvowvRzK0EgBSpE+fPlq1apXWrl3rdygZo3v37urTp4/fYQAA\nMgj50tPVHEkhBwApUlhYqH79+vkdBgAAGY18GR+GVgIAAABAlqGQAwAAAIAsQyEHAAAAAFnGMnl1\nGDNbK2llGm/ZQ9JXabxfOuXqe8vV9yXx3rIV7y0+ezvneqbotXOOD/kxU+Xy37dE8H2Jju9NdHxv\novP7e9NujszoQi7dzGyec67M7zhSIVffW66+L4n3lq14b0D68DvZPr4v0fG9iY7vTXSZ+r1haCUA\nAAAAZBkKOQAAAADIWCEgugAABUJJREFUMhRykSb7HUAK5ep7y9X3JfHeshXvDUgffifbx/clOr43\n0fG9iS4jvzfMkQMAAACALEOPHAAAAABkmbwu5MzsLDN7z8yazCzqSjRmtsLMFpnZQjObl84Y49GF\n93WSmS01s2VmNjadMcbLzHYxsxfN7KPgnztHua4x+PNaaGbT0h1nV3T2czCzbcxsSvDxOWbWN/1R\nxieG9zbSzNaG/axG+RFnV5nZfWa2xswWR3nczOz24Pt+18wOTneM8YrhvR1jZhvDfmbj0x0j8lcu\n57dE5GJuTFQu59ZE5WpuTlQ25va8LuQkLZY0XNKrMVz7E+fc4ExcerQdnb4vMyuQdJekkyXtL+lc\nM9s/PeElZKykmc65/pJmBtvtqQv+vAY7585IX3hdE+PP4ReSNjjnfijpVkk3pTfK+HThd2xK2M/q\n3rQGGb8HJJ3UweMnS+of/Bot6e40xJQsD6jj9yZJr4X9zP6UhpiAZrmc3xKRU7kxUbmcWxOV47k5\nUQ8oy3J7XhdyzrklzrmlfseRbDG+ryGSljnnljvntkh6XNKw1EeXsGGSHgwePyipwsdYkiGWn0P4\ne35S0nFmZmmMMV7Z+jvWKefcq5LWd3DJMEkPOc9bkorNbI/0RJeYGN4b4Jscz2+JyLXcmKhczq2J\nyse/HzHJxtye14VcFzhJL5jZfDMb7XcwSVIi6bOw9qrguUzXyzn3efD4C0m9olzX3czmmdlbZpbJ\nCS2Wn0PLNc65BkkbJe2alugSE+vv2E+DQxSeNLM90xNaymXr369YHWFm75jZc2Z2gN/BAK3k+t+/\n9uRabkxULufWROVzbk5Uxv3bspWfN08HM3tJ0u7tPHStc+7pGF/mKOdcwMx2k/SimX0QrNp9k6T3\nlZE6em/hDeec+//t3TFrFEEUwPH/A1FBRNQUKoIYEGyFIBI7EYsUAcHCyhQpTOGnsBH8AFbpU1go\nEQKCREtREEJQQY2VQRK0UGyCxVjsHByG5CJ73t7s/X+w3GQvLO/t3OZl9ubmImKnZVfP5D4bB5Yj\nYjWltNbvWFXbE2AhpbQVEbep7o5eaTgm7e4N1fX1KyKmgMdU00ykvmhzfavD2qgBsjYXovUDuZTS\n1T4cYz0/bkbEI6q3pRsdyPUhr3Wg+w7L6byvcbvlFhEbEXEypfQ1v529ucMxOn32OSJeABeAYSxW\ne+mHzu98iYh9wBHg+2DCq6Vnbiml7jzmgfsDiGsQhvb6qiul9LOrvRQRDyJiLKX0rcm41B5trm91\njFhtrKvNtbWuUa7NdQ3d3xanVvYQEYci4nCnDVyj+rB16V4D5yLibETsB24CJaxgtQjM5PYMsO3u\nbEQcjYgDuT0GXAbeDSzCf7OXfujO+QawnMr4Asieuf01t3waeD/A+P6nReBWXuHqEvCja9pT0SLi\nROdzJBFxkaqOjMI/PypHqfWtjrbVxrraXFvrGuXaXNfw1faU0shuwHWq+a1bwAbwNO8/BSzl9jiw\nkre3VFM7Go+9bl755yngA9XduKHPK8d8nGpFro/AM+BY3j8BzOf2JLCa+2wVmG067h45besH4C4w\nndsHgYfAJ+AVMN50zH3M7V6+rlaA58D5pmPeY14LwFfgd77WZoE5YC4/H1Srgq3l1+BE0zH3Mbc7\nXX32EphsOma30dnaXN9qnpfW1cY+nJPW1tYBnJsia3MfzktxtT1yYJIkSZKkQji1UpIkSZIK40BO\nkiRJkgrjQE6SJEmSCuNATpIkSZIK40BOkiRJkgrjQE6SJEmSCuNATpIkSZIK40BOkiRJkgrzB4qD\nNewjez9dAAAAAElFTkSuQmCC\n","text/plain":["<Figure size 1080x360 with 2 Axes>"]},"metadata":{"tags":[]}}]},{"cell_type":"markdown","metadata":{"id":"e9ppP7yDmAq7","colab_type":"text"},"source":["## Inference"]},{"cell_type":"markdown","metadata":{"id":"x1EeMpH8mD3f","colab_type":"text"},"source":["After training a model, we can use it to predict on new data."]},{"cell_type":"code","metadata":{"id":"gyGJ3ossi6Gm","colab_type":"code","colab":{}},"source":["# Feed in your own inputs\n","sample_indices = [10, 15, 25]\n","X_infer = np.array(sample_indices, dtype=np.float32)\n","X_infer = X_scaler.transform(X_infer.reshape(-1, 1))"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"9PuOcgwlmMST","colab_type":"text"},"source":["Recall that we need to unstandardize our predictions.\n","\n","$ \\hat{y}_{scaled} = \\frac{\\hat{y} - \\mu_{\\hat{y}}}{\\sigma_{\\hat{y}}} $\n","\n","$ \\hat{y} = \\hat{y}_{scaled} * \\sigma_{\\hat{y}} + \\mu_{\\hat{y}} $"]},{"cell_type":"code","metadata":{"id":"j0uQOCzai5-G","colab_type":"code","outputId":"7ce41969-17ea-4200-9e55-12876ce5fbb0","executionInfo":{"status":"ok","timestamp":1583941135121,"user_tz":420,"elapsed":555,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":68}},"source":["# Unstandardize predictions\n","pred_infer = model.predict(X_infer) * np.sqrt(y_scaler.var_) + y_scaler.mean_\n","for i, index in enumerate(sample_indices):\n","    print (f\"{df.iloc[index]['y']:.2f} (actual) → {pred_infer[i][0]:.2f} (predicted)\")"],"execution_count":159,"outputs":[{"output_type":"stream","text":["35.73 (actual) → 43.36 (predicted)\n","59.34 (actual) → 60.37 (predicted)\n","97.04 (actual) → 94.40 (predicted)\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"1TMS8GvcmWZ0","colab_type":"text"},"source":["## Interpretability"]},{"cell_type":"markdown","metadata":{"id":"jd_eIgQBmYzm","colab_type":"text"},"source":["Linear regression offers the great advantage of being highly interpretable. Each feature has a coefficient which signifies its importance/impact on the output variable y. We can interpret our coefficient as follows: by increasing X by 1 unit, we increase y by $W$ (~3.65) units. "]},{"cell_type":"code","metadata":{"id":"YvIKQ6sVmV-A","colab_type":"code","outputId":"1416196c-c052-4fa3-90bb-408650c1b5ce","executionInfo":{"status":"ok","timestamp":1583941140253,"user_tz":420,"elapsed":356,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":51}},"source":["# Unstandardize coefficients \n","W = model.layers[0].get_weights()[0][0][0]\n","b = model.layers[0].get_weights()[1][0]\n","W_unscaled = W * (y_scaler.scale_/X_scaler.scale_)\n","b_unscaled = b * y_scaler.scale_ + y_scaler.mean_ - np.sum(W_unscaled*X_scaler.mean_)\n","print (\"[actual] y = 3.5X + noise\")\n","print (f\"[model] y_hat = {W_unscaled[0]:.1f}X + {b_unscaled[0]:.1f}\") "],"execution_count":160,"outputs":[{"output_type":"stream","text":["[actual] y = 3.5X + noise\n","[model] y_hat = 3.4X + 9.3\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"7-BrAEsOsj3g","colab_type":"text"},"source":["# Regularization"]},{"cell_type":"markdown","metadata":{"id":"rCEbjZARsmBc","colab_type":"text"},"source":["Regularization helps decrease overfitting. Below is L2 regularization (ridge regression). There are many forms of regularization but they all work to reduce overfitting in our models. With L2 regularization, we are penalizing the weights with large magnitudes by decaying them. Having certain weights with high magnitudes will lead to preferential bias with the inputs and we want the model to work with all the inputs and not just a select few. There are also other types of regularization like L1 (lasso regression) which is useful for creating sparse models where some feature cofficients are zeroed out, or elastic which combines L1 and L2 penalties. \n","\n","**Note**: Regularization is not just for linear regression. You can use it to regularize any model's weights including the ones we will look at in future lessons."]},{"cell_type":"markdown","metadata":{"id":"QfVSaAQfspyk","colab_type":"text"},"source":["$ J(\\theta) = = \\frac{1}{2}\\sum_{i}(X_iW - y_i)^2 + \\frac{\\lambda}{2}W^TW$\n","\n","$ \\frac{\\partial{J}}{\\partial{W}}  = X (\\hat{y} - y) + \\lambda W $\n","\n","$W = W- \\alpha\\frac{\\partial{J}}{\\partial{W}}$\n","* $\\lambda$ is the regularzation coefficient"]},{"cell_type":"code","metadata":{"id":"83jtd_u2sli1","colab_type":"code","colab":{}},"source":["from tensorflow.keras.regularizers import l2"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"E-KcdfC1slrn","colab_type":"code","colab":{}},"source":["L2_LAMBDA = 1e-2"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"QjvyB0_vaGHw","colab_type":"code","colab":{}},"source":["class L2LinearRegression(Model):\n","    def __init__(self, l2_lambda, output_dim):\n","        super(L2LinearRegression, self).__init__(name='l2_linear_regression')\n","        self.fc1 = Dense(units=output_dim, activation='linear', \n","                         kernel_regularizer=l2(l=l2_lambda), name='W')\n","        \n","    def call(self, x_in, training=False):\n","        y_pred = self.fc1(x_in)\n","        return y_pred\n","    \n","    def summary(self, input_shape):\n","        x_in = Input(shape=input_shape, name='X')\n","        summary = Model(inputs=x_in, outputs=self.call(x_in), name=self.name)\n","        return plot_model(summary, show_shapes=True) # forward pass"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"-yBiOgL-slpy","colab_type":"code","outputId":"053615a1-4d16-458b-ed67-d97901a4379a","executionInfo":{"status":"ok","timestamp":1583941144835,"user_tz":420,"elapsed":1012,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":201}},"source":["# Initialize model\n","model = L2LinearRegression(l2_lambda=L2_LAMBDA, output_dim=OUTPUT_DIM)\n","model.summary(input_shape=(INPUT_DIM,))"],"execution_count":164,"outputs":[{"output_type":"execute_result","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAASUAAAC4CAIAAAAqim0JAAAABmJLR0QA/wD/AP+gvaeTAAAePklE\nQVR4nO3dezxU6/448GcxMUbjVkikXNJOoZuOEXU62lSK3DKls4961XHZ56Ccs0W1k6LaeuFITq86\nqr27UfRCRe2tV3Y5O3YlEju3tkoqJJcYjLF+fzzf73znh2qMmTUXn/dfrWeteZ5nzfi0bs+zPgRJ\nkggAQAklaXcAgHEE4g0A6kC8AUAdiDcAqEMTXLh3715CQoK0ugKA4mGxWDt27OAv/n/Ht5cvX2Zm\nZlLeJTBUcXFxcXGxtHshcY2NjYr991ZcXHzv3j3BEtrwjS5fvkxVf8DIfHx80Dj4IS5duuTr66vA\nu4l/R0Fw/QYAdSDeAKAOxBsA1IF4A4A6EG8AUAfiTXHk5eVpampevXpV2h0Rs8DAQOJ/bdq0SXBV\nQUFBZGTk4OCgh4eHsbExnU43NDR0d3d//Pix8PUPDg4mJiba29sLFubm5h4+fJjH4/FLsrOz+d2Y\nPHmyaPsC8aY4FHiqh46OTn5+fnV1dVpaGr9w7969ycnJUVFRg4ODd+/evXDhQltbW1FREYfDWbp0\naVNTkzA119bWLl26dMeOHT09PYLlbm5udDrdycmpvb0dl7i7uzc2Nt65c2f16tUi7wjEm+JwdXXt\n6OhYu3atpBvicDhDjgaSpqamtnLlSgsLC1VVVVxy6NCh9PT0S5cuMZlMhBCLxXJwcGAwGCYmJrGx\nsR0dHWfOnPlsteXl5Tt37gwKCpo3b97wtaGhoTY2NqtXrx4YGEAIEQRhaGjo6Og4c+ZMkXcE4g2M\nWlpaWnNzsxQ7UFdXt2fPnn379tHpdIQQjUYTPIs2NTVFCNXX13+2Hhsbm6ysLD8/P34YDxEdHV1W\nVpaUlCSmjkO8KYqioiJjY2OCIFJSUhBCqamp6urqDAYjJydn1apVGhoaRkZGFy9exBsnJyfT6XQ9\nPb3AwEADAwM6nW5vb19SUoLXhoSEqKioTJkyBS9+/fXX6urqBEG0trYihMLCwsLDw+vr6wmCMDc3\nRwjduHFDQ0MjNjaWsp1NTk4mSdLNzW3EtRwOByGkoaEx9oa0tbWXLVuWlJQkrnN1iDcF4eDg8Msv\nv/AXg4ODt2/fzuFwmExmRkZGfX29qanptm3buFwuQigkJMTf37+npyc0NLShoaG0tHRgYODLL798\n+fIlQig5OXn9+vX8qo4dO7Zv3z7+YlJS0tq1a83MzEiSrKurQwjhmwqDg4OU7ez169dnzZrFYDBG\nXPvrr78ihBwcHMTS1vz581+9elVeXi6W2iDeFJy9vb2Ghoauri6bze7u7n7x4gV/FY1Gmz17tqqq\nqqWlZWpqaldX1+nTp0VowtXVtbOzc8+ePeLr9ad0d3f//vvvZmZmw1e9ffs2PT09NDSUxWJ97Og3\nWvhqraKiQiy1jTBeGSgkFRUVhBA+vg23aNEiBoPx9OlTajsliubmZpIkRzy4sVis7u7u9evXHzhw\nYMKECWJpDjf09u1bsdQG8Qb+h6qqaktLi7R78Xm9vb0IoRHvcOjp6aWlpc2ZM0eMzampqfEbHTs4\nnwQIIcTlctvb242MjKTdkc/DASD4JJpPV1dXS0tLvM319/fzGx07OL4BhBAqLCwkSdLOzg4v0mi0\nj515Sp2enh5BEB0dHcNXSWJsDW5IX19fLLXB8W38GhwcfP/+/cDAwOPHj8PCwoyNjf39/fEqc3Pz\ntra27OxsLpfb0tLy/PlzwQ/q6Og0NTU1NDR0dXVxudz8/HwqnwcwGAxTU9PGxsYh5XV1dfr6+r6+\nvoKFbDZbX1+/tLRU5OZwQ1ZWViLXIAjiTUGkpKTY2toihCIiItzd3VNTUxMTExFC1tbWz549O3ny\nZHh4OEJo5cqVtbW1+CO9vb1WVlZqamqOjo4WFha3b9/mXxQFBwcvX758w4YNs2bN2r9/Pz6bYrFY\n+IFBUFCQnp6epaXl6tWr29raqN9ZV1fXyspK/JyNb8RHZP39/c3NzTk5OSPWU1xc7ODgMHXq1JKS\nkvLycgMDgyVLlty5c0dwm/v37xsaGlpbW4un66SAjIyMISVAKry9vb29vSXaREBAgI6OjkSb+Cwh\n/94CAgIMDQ0FS2pra2k02tmzZz/7WR6P5+jomJaWJloPW1tb6XT6kSNHBAtDQ0MnTZokzMeH/45w\nfBu/RrzlIJs4HM7Nmzdra2vx3Qtzc/OYmJiYmJgPHz584lM8Hi87O7urq4vNZovWbnR09Lx580JC\nQhBCJEk2NTUVFRXhp/yigXgDcqCtrQ2PV96yZQsuiYyM9PHxYbPZI944wQoLC7OysvLz8z82EuXT\nEhISysrK8vLy8KO8nJwcPF75+vXrou0FEi3e7O3tJ0yYQBCEiorKggUL3rx5gxA6c+aMkZERnhqU\nmpr62UqKi4tnz56tpKREEIS+vv6BAwdE6IlosrKyTE1N8USmKVOmDJlSNR5ERUWdPn26o6PDxMRE\n9t9Id/z4cf752Llz5/jlsbGxISEhBw8e/NgHnZyczp8/zx8IOio5OTl9fX2FhYXa2tq4ZN26dYLn\nmSLUiZCo128FBQV4uGp3dze/8MKFC4sXL+7v7xemBszFxQUh9P79e+E/Ii5mZmaamprUtysMCq7f\nZIHC3y8Q2/Wbk5NTYGBgXV3dzp07cUlNTU1ERERGRoa4xtGIF/VTtgAYTvTrt/j4eFNT05SUlMLC\nQg6H4+Pjc/To0RkzZoivb+Ik9SlbAKCxxJu6ujoeTr5ly5a//vWvTk5O7u7uQ7YZ1cwoKqdsCePu\n3buWlpaampp0Ot3KyurmzZsIoa1bt+ILPzMzs0ePHiGENm/ezGAwNDU1c3NzEUI8Hu/bb781NjZW\nU1OztrbGp0zfffcdg8FgMpnNzc3h4eGGhobV1dVCdgMoFMGTSxHOp0NDQxFCJiYmI162Xbt2jclk\nxsTEfOzjQ67fdu3ahRC6detWR0dHc3Ozo6Ojuro6v+aAgAB1dfWqqqre3t7KykpbW1smk/nixQu8\n1s/PT19fn19zfHw8QqilpQUvenl54SlbfJ+9frt8+XJ0dHRbW9u7d+/s7Oz4j1y8vLyUlZVfvXrF\n33Ljxo25ubn43//4xz9UVVUzMzPfv38fFRWlpKR0//59/q6FhoYePXrU09Pzt99++0TTcP2mGMT/\n/A3fn21oaLh79+7wtaLNjKJgypYwvL299+7dq62traOj4+bm9u7dOzx8PigoiMfj8dvt7Oy8f/8+\nfodMb29vamqqh4eHl5eXlpbW7t27J0yYINjDQ4cO/e1vf8vKyvriiy8k1G0gy8YUb319fZs3b96z\nZ4+SktKWLVu6urrE1S1MdqZs4ZtA+AHxn/70JwsLi1OnTpEkiRBKT09ns9nKysoIoerq6p6enrlz\n5+JPqampTZkyRbQeZmZmEooOj3WUdi8kaPizljHND9i+ffuyZctiYmL6+/sPHz4cHh5+4sSJsVQ4\nWhKdsnX9+vX4+PjKysrOzk7BmCcIIjAwcMeOHbdu3VqxYsUPP/xw/vx5vKq7uxshtHv37t27d/O3\nNzAwEKF1Ozu77du3j20PZN29e/eSkpLwWaVCwkNYBYkeb5cuXXr48GFRURFCaN++fdeuXTt58qSn\np+fKlSvH1EehSWLK1p07dx4+fLh9+/YXL154eHh4enqeOnVq6tSpR48e/eabb/ib+fv7R0VF/ec/\n/5k2bZqGhsb06dNxua6uLkIoMTExLCxsjD0xMjISfImIokpKSlLg3RyeakvEeKuvr//mm28KCwvx\niZaqqur3339vZ2e3devWJ0+eiH3O34gkMWXr4cOH6urqCKGKigoulxscHIxfrkYQhOBm2travr6+\n6enpTCZz27Zt/PJp06bR6fSysrIxdgMoKlGu3/r6+nx9fZOSkgSfti1cuDAyMvLVq1f4jiUm9plR\n4pqyNbxmLpf79u3bwsJCHG/GxsYIoYKCgt7e3traWv6DB76goKC+vr5r164Jvl+VTqdv3rz54sWL\nqampnZ2dPB6vsbHx9evX4tp9IPcEb1YKc382JSUFvzx9xowZ6enp/PIdO3ZMmjQJ12liYlJQUECS\nZF5eHpPJPHDgwPB6iouL58yZo6SkhBCaMmVKbGzssWPH8LjSmTNn1tfXnzhxAr9CcPr06TU1NSRJ\nBgQETJgwwdDQkEajaWhorFu3rr6+nl/hu3fvli9fTqfTTUxM/v73v//zn/9ECJmbm+MHBqWlpdOn\nT1dTU3NwcPj3v/894tudsCtXruAKIyIidHR0tLS0fHx88EsdzczM+I8fSJKcP39+ZGTkkP3q6+uL\niIgwNjam0Wi6urpeXl6VlZWHDx/GU8imTZsmzCwSeB6gGIb/jvI0/00WpmwJWr169bNnzyRRM8Sb\nYpD7+W9Sn7LFPxd9/PgxPpZKtz9AvshZvEldREREbW1tTU3N5s2b9+/fL+3ujAuQj0oKZGTKFoPB\n+OKLL1asWBEdHW1paSmtbow3CpOPSp6u38YPCq7fenp6WCyWdKsS+f0lJEkePHjQwsKCw+GQJMnl\nctesWcNfhfMHxMbGfrbmsrIyT0/Pc+fOzZs3z8bGZvgGISEhLBaLy+UKFsL7S8CoiXGCEvVznSAf\nFZACkiQTEhLwAG5tbe1169bxx2qOaoKS3KWngnxUQAqio6MjIyN37drV3Nx8586dly9fOjo64swS\no8opJXfpqSAfFaAah8NJSEjw9PTctGmTpqamlZXV8ePHW1tbRR4yLi/pqSAfFZCCysrKDx8+LFq0\niF9ia2uroqIyfOiZCGQ5PRXkowJSgO9TT5w4UbBQS0tLXLMQZTY9FeSjAlKAJ2EMiS5xTVCS5fRU\nkI8KSMHcuXMnTpz44MEDfklJSUl/f//ChQvx4lgmKMlyeirIRwWkgE6nh4eHX7ly5dy5c52dnRUV\nFUFBQQYGBgEBAXiD0U5Qkpf0VJCPCkjH3r174+LiYmJiJk+evGzZshkzZvDn76HR55SSo/RUkI8K\niBP183GkMtcJ8lGB8Uvqc50+AfJRAUCdcZ2PCigYGZnr9DGKlI8KngcAFBcXFxcXJ+1eiMLZ2dnZ\n2VkSNbu7uw/PhzF2cHwDgDoQbwBQB+INAOpAvAFAnRHul1y6dIn6fgBBeAyRwv8Q9+7dQwq9m42N\njUPHfAs+/FbgTCUASMWQ8SUEKaYXMwAqEQSRkZGhwJllFBVcvwFAHYg3AKgD8QYAdSDeAKAOxBsA\n1IF4A4A6EG8AUAfiDQDqQLwBQB2INwCoA/EGAHUg3gCgDsQbANSBeAOAOhBvAFAH4g0A6kC8AUAd\niDcAqAPxBgB1IN4AoA7EGwDUgXgDgDoQbwBQB+INAOpAvAFAHYg3AKgD8QYAdSDeAKAOxBsA1IF4\nA4A6EG8AUAfiDQDqQLwBQB3IbyofAgICqqur+YulpaUmJiba2tp4UVlZ+fvvvx+aKhrIHpq0OwCE\noq+vf+LECcGSx48f8/9tamoKwSYX4HxSPmzcuPFjq1RUVPz9/SnsCxAdnE/Kjblz51ZVVY34e1VX\nV1tYWFDfJTBacHyTG1999ZWysvKQQoIgbGxsINjkBcSb3NiwYQOPxxtSqKys/Je//EUq/QEigPNJ\neWJvb19SUjI4OMgvIQji5cuXhoaGUuwVEB4c3+TJn//8Z4Ig+ItKSkoODg4QbHIE4k2e+Pj4CC4S\nBPHVV19JqzNABBBv8mTy5MlOTk78uyYEQXh4eEi3S2BUIN7kzKZNm/Alt7KysouLy6RJk6TdIzAK\nEG9yxtPTU0VFBSFEkuSmTZuk3R0wOhBvckZdXX3NmjUIIRUVlbVr10q7O2B0IN7kj5+fH0LIw8ND\nXV1d2n0Bo0RKgLT3CQAxyMjIEHtoSGp+QFhYGIvFklDl8s7X13eM38+5c+fYbDaNJtPTOxITExFC\n27dvl3ZHROHr6yuJaiUyvoQgiIyMjPXr14u9ZsUw9u+nt7eXTqeLsUuSgJ8WXr58WdodEYWE/obh\n+k0uyX6wgRFBvAFAHYg3AKgD8QYAdSDeAKAOxJvcyMvL09TUvHr1qrQ7IikFBQWRkZGDg4MeHh7G\nxsZ0Ot3Q0NDd3V3wzUifNTg4mJiYaG9vL1iYm5t7+PDh4bN1qQfxJjcUeyDB3r17k5OTo6KiBgcH\n7969e+HChba2tqKiIg6Hs3Tp0qamJmEqqa2tXbp06Y4dO3p6egTL3dzc6HS6k5NTe3u7ZLovLIg3\nueHq6trR0UHBmEkOhzPk+CBphw4dSk9Pv3TpEpPJRAixWCwHBwcGg2FiYhIbG9vR0XHmzJnPVlJe\nXr5z586goKB58+YNXxsaGmpjY7N69eqBgQGx9194EG9gqLS0tObmZsqaq6ur27Nnz759+/BDRRqN\nJnjObGpqihCqr6//bD02NjZZWVl+fn6qqqojbhAdHV1WVpaUlCSmjosC4k0+FBUVGRsbEwSRkpKC\nEEpNTVVXV2cwGDk5OatWrdLQ0DAyMrp48SLeODk5mU6n6+npBQYGGhgY0Ol0/OITvDYkJERFRWXK\nlCl48euvv1ZXVycIorW1FSEUFhYWHh5eX19PEIS5uTlC6MaNGxoaGrGxsRLateTkZJIk3dzcRlzL\n4XAQQhoaGmNvSFtbe9myZUlJSVI8M4d4kw8ODg6//PILfzE4OHj79u0cDofJZGZkZNTX15uamm7b\nto3L5SKEQkJC/P39e3p6QkNDGxoaSktLBwYGvvzyy5cvXyKEkpOTBYcpHTt2bN++ffzFpKSktWvX\nmpmZkSRZV1eHEMK3GQRfUiRe169fnzVrFoPBGHHtr7/+ihBycHAQS1vz589/9epVeXm5WGoTAcSb\nfLO3t9fQ0NDV1WWz2d3d3S9evOCvotFos2fPVlVVtbS0TE1N7erqOn36tAhNuLq6dnZ27tmzR3y9\n/j/d3d2///67mZnZ8FVv375NT08PDQ1lsVgfO/qN1syZMxFCFRUVYqlNBDI9wBwID0/6xse34RYt\nWsRgMJ4+fUptpz6vubmZJMkRD24sFqu7u3v9+vUHDhyYMGGCWJrDDb19+1YstYkA4m28UFVVbWlp\nkXYvhurt7UUIjXiHQ09PLy0tbc6cOWJsTk1Njd+oVMD55LjA5XLb29tlMIcODoARn0Tr6upqaWmJ\nt7n+/n5+o1IBx7dxobCwkCRJOzs7vEij0T525kkxPT09giA6OjqGr5LESBrckL6+vthrFhIc3xTW\n4ODg+/fvBwYGHj9+HBYWZmxszE9bZW5u3tbWlp2dzeVyW1panj9/LvhBHR2dpqamhoaGrq4uLpeb\nn58vuecBDAbD1NS0sbFxSHldXZ2+vv6QSdZsNltfX7+0tFTk5nBDVlZWItcwRhBv8iElJcXW1hYh\nFBER4e7unpqait9WYG1t/ezZs5MnT4aHhyOEVq5cWVtbiz/S29trZWWlpqbm6OhoYWFx+/Zt/mVS\ncHDw8uXLN2zYMGvWrP379+PzKxaLhR8YBAUF6enpWVparl69uq2tTdK75urqWllZiZ+z8Y34iKy/\nv7+5uTknJ2fEeoqLix0cHKZOnVpSUlJeXm5gYLBkyZI7d+4IbnP//n1DQ0Nra2sx9n90xP5GFPxN\nSeJdKwqDgu8nICBAR0dHok18lre3t7e392c3q62tpdFoZ8+e/eyWPB7P0dExLS1NtP60trbS6fQj\nR44Is7GEfiM4viksWRgOLwxzc/OYmJiYmJgPHz58YjMej5ednd3V1cVms0VrKDo6et68eSEhIaJ9\nXCykE28LFiwgCIIgCAMDg9DQ0I9tVlNTY2trO3HiRCUlpZUrV36iwqysLFNTU0KAioqKnp7eH//4\nx/j4+Pfv30tgJ4DYREZG+vj4sNnsEW+cYIWFhVlZWfn5+R8bifJpCQkJZWVleXl54nqUJyKxHzFJ\n4Y7Fzs7OBEG8fv1asHBgYGD58uVDtvzXv/7l5+cnTLtmZmaampokSeJbBbdv3/b398dRff/+/dHs\ngWQJ8/2MRWRkJH78PWPGjMuXL0uuoU8T8nyS7+bNmxEREZLoSXZ2dlxc3MDAgPAfkdBvJLV4O3Xq\nFELo5MmTgoU//fQTQqiyslKw0MXFJTc3V5h2+fEm6PLly0pKSnp6eu3t7cJUQgFJx5uMGG28yRQJ\n/UZSu37DeSdyc3MFC3/88cepU6dmZmbySzgcTnl5uYuLi8gNeXt7+/v7Nzc3Hz9+XPTuAiAOUos3\nTU1NFxeXgoIC/o1g/ATW29tbMN5u3brl4uKCz45EnhiCnzvl5+fjRR6P9+233xobG6upqVlbW2dk\nZKDPzXBBCP3888+LFy9mMBgaGhpWVladnZ0fqwqAj5Hm/Uk2m83hcAoKCvDiTz/9tGLFCh8fn4qK\nipqaGlyYl5fHf+gp8sQQPOH32bNneHHnzp3fffddYmLi69ev165du3HjxgcPHnx6hkt3d7ebm5u3\nt3dbW1ttba2FhQUeGTRiVWP9XoACE/sZKin0uW9XV5eamtrWrVvxYnh4OJfLHRwcnDp16oEDB3Dh\nggUL+vv7hWx3xOs3jCAILS0tkiQ5HA6DwWCz2bi8p6dHVVU1ODiYJMldu3YhhDgcDl517NgxhFBd\nXR1Jkk+ePEEIXbt2TbDOT1T1aUJ+P/IOrt+Gk+b4yYkTJ7q6uuI/4v7+fhqNhhNQeHl5ZWZm7tq1\nq6qqav78+WO/gdvd3U2SJJ4jXF1d3dPTM3fuXLxKTU1typQpI05UEZzhYmpqqqent2nTptDQUH9/\n/xkzZoyqquHu3bs3xp2SfXjw1KVLl6TdEVki9ggmR/N/A75UKykpuXLlyu3bt3Hhzz//jBCqq6uL\nj4//8ccfhW/3Y8c3POLO2dmZJMn//ve/w78EOzs7ctjx7eTJkwih3377DS8+efJkzZo1NBqNIAhf\nX9+enp5PVPVpkvw9gdgo1P1JzNXVlclk5ubm3rlzZ+nSpbjQwcHBwMAgMzPzwYMHy5cvH3srN27c\nQAitWrUKIaSrq4sQSkxMFPwWhDnazJkz5+rVq01NTRERERkZGUeOHBG5KgTnkzJvTH9wHyfleKPT\n6W5ubpmZmWpqakpK/9MZJSUlT0/PH374QU9Pb+wpzt68eZOYmGhkZLRlyxaE0LRp0+h0ellZ2agq\naWpqqqqqQgjp6uoePHhwwYIFVVVVolUFxjPpj59ks9nV1dU4JzWfj49PVVWVh4eHYKEwE0NIkvzw\n4cPg4CBJki0tLRkZGUuWLFFWVs7OzsbXb3Q6ffPmzRcvXkxNTe3s7OTxeI2Nja9fv/50J5uamgID\nA58+fdrf3//o0aPnz5/b2dmJVhUY1yR0LBb+fKm/v9/GxgZHCB+Px7OxseHxeIKFeXl5TCaTf+tS\nUG5urrW1NYPBUFFRwcdJfENy8eLFMTEx7969E9y4r68vIiLC2NiYRqPp6up6eXlVVlYeO3YMD8yb\nOXNmfX39iRMncHxOnz69pqamoaHB3t5eW1tbWVl56tSpu3btwoODRqxKvN+P/JL380lJ/EaQ31QK\nxsn3A/lNh5P++SQA4wfEGwDUgXgDsgLyUQFAEchHBeSVGBNKUZObCvJRATkmxoRSFOSmgnxUQPpI\nkkxISMAJN7S1tdetW8cfCT2qhFKyn5tqXOWjkv7z7nFImO/n22+/VVFROXv2bHt7++PHjxcsWDB5\n8uQ3b97gtX5+fvr6+vyN4+PjEUItLS140cvLCyeUwgICAtTV1auqqnp7eysrK21tbZlM5osXL0So\n6tq1a0wmMyYmRpjdFPJ5t6mpqaWl5cfWZmVlIYQyMzOFaRH7wx/+YGNjM+KqyMhIhNCjR48+W4mE\n/obh+CaLOBxOQkKCp6fnpk2bNDU1raysjh8/3traeuLECdEqlNncVJCPCkhfZWXlhw8fFi1axC+x\ntbVVUVHhnweOhUzlpoJ8VED68G3riRMnChZqaWl1dXWJpX7ZyU0F+aiA9OE8TEOiS1wJpWQqNxXk\nowLSN3fu3IkTJwq+eqikpKS/v3/hwoV4cSwJpWQqNxXkowLSR6fTw8PDr1y5cu7cuc7OzoqKiqCg\nIAMDg4CAALzBqBJKIRnOTQX5qIBM2Lt3b1xcXExMzOTJk5ctWzZjxozCwkJ1dXW8drQJpWQ5NxXk\noxorBM/fPoni70dauakgH9VwcHwbF2RhaPzHQD4qACg1fvJRQbwpuKioqNOnT3d0dJiYmAgmZpA1\nsbGxISEhBw8e/NgGTk5O58+f5w/1HJWcnJy+vr7CwkJtbe0x9FEM4HmAgouLi4uLi5N2L4Ti7Ozs\n7OwsiZrd3d3d3d0lUfNowfENAOpAvAFAHYg3AKgD8QYAdSR1vyQxMVFOX/RJjfHw/RQXF6P/fesr\nwCTyfmX4ioEC2LFjB4vFEm+dEok3AMCI4PoNAOpAvAFAHYg3AKgD8QYAdf4fdiNZs/TpkOAAAAAA\nSUVORK5CYII=\n","text/plain":["<IPython.core.display.Image object>"]},"metadata":{"tags":[]},"execution_count":164}]},{"cell_type":"code","metadata":{"id":"l7WatvzFtINs","colab_type":"code","colab":{}},"source":["# Compile\n","model.compile(optimizer=Adam(lr=LEARNING_RATE),\n","              loss=MeanSquaredError(),\n","              metrics=[MeanAbsolutePercentageError()])"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"3K7lVyQStILO","colab_type":"code","outputId":"095306cc-a43a-44a0-e44d-e2c2c319b3db","executionInfo":{"status":"ok","timestamp":1583941147100,"user_tz":420,"elapsed":1598,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":1000}},"source":["# Training\n","model.fit(x=X_train, \n","          y=y_train,\n","          validation_data=(X_val, y_val),\n","          epochs=NUM_EPOCHS,\n","          batch_size=BATCH_SIZE,\n","          shuffle=SHUFFLE,\n","          verbose=1)"],"execution_count":166,"outputs":[{"output_type":"stream","text":["Train on 35 samples, validate on 7 samples\n","Epoch 1/50\n","35/35 [==============================] - 0s 9ms/sample - loss: 0.3463 - mean_absolute_percentage_error: 64.7264 - val_loss: 0.0935 - val_mean_absolute_percentage_error: 132.8669\n","Epoch 2/50\n","35/35 [==============================] - 0s 430us/sample - loss: 0.0612 - mean_absolute_percentage_error: 28.1441 - val_loss: 0.0314 - val_mean_absolute_percentage_error: 882.2163\n","Epoch 3/50\n","35/35 [==============================] - 0s 451us/sample - loss: 0.0474 - mean_absolute_percentage_error: 27.2072 - val_loss: 0.0277 - val_mean_absolute_percentage_error: 866.8756\n","Epoch 4/50\n","35/35 [==============================] - 0s 358us/sample - loss: 0.0819 - mean_absolute_percentage_error: 36.9764 - val_loss: 0.0243 - val_mean_absolute_percentage_error: 359.9240\n","Epoch 5/50\n","35/35 [==============================] - 0s 394us/sample - loss: 0.0596 - mean_absolute_percentage_error: 34.5615 - val_loss: 0.0262 - val_mean_absolute_percentage_error: 682.9621\n","Epoch 6/50\n","35/35 [==============================] - 0s 401us/sample - loss: 0.0370 - mean_absolute_percentage_error: 25.2826 - val_loss: 0.0396 - val_mean_absolute_percentage_error: 922.7436\n","Epoch 7/50\n","35/35 [==============================] - 0s 406us/sample - loss: 0.0383 - mean_absolute_percentage_error: 24.8166 - val_loss: 0.0468 - val_mean_absolute_percentage_error: 588.8998\n","Epoch 8/50\n","35/35 [==============================] - 0s 412us/sample - loss: 0.0424 - mean_absolute_percentage_error: 26.9451 - val_loss: 0.0457 - val_mean_absolute_percentage_error: 533.3515\n","Epoch 9/50\n","35/35 [==============================] - 0s 397us/sample - loss: 0.0383 - mean_absolute_percentage_error: 26.8012 - val_loss: 0.0374 - val_mean_absolute_percentage_error: 558.5472\n","Epoch 10/50\n","35/35 [==============================] - 0s 589us/sample - loss: 0.0341 - mean_absolute_percentage_error: 25.9799 - val_loss: 0.0312 - val_mean_absolute_percentage_error: 716.1393\n","Epoch 11/50\n","35/35 [==============================] - 0s 445us/sample - loss: 0.0336 - mean_absolute_percentage_error: 25.6114 - val_loss: 0.0284 - val_mean_absolute_percentage_error: 545.0917\n","Epoch 12/50\n","35/35 [==============================] - 0s 430us/sample - loss: 0.0371 - mean_absolute_percentage_error: 28.5534 - val_loss: 0.0287 - val_mean_absolute_percentage_error: 401.7839\n","Epoch 13/50\n","35/35 [==============================] - 0s 515us/sample - loss: 0.0340 - mean_absolute_percentage_error: 26.2566 - val_loss: 0.0334 - val_mean_absolute_percentage_error: 804.1356\n","Epoch 14/50\n","35/35 [==============================] - 0s 441us/sample - loss: 0.0346 - mean_absolute_percentage_error: 25.6177 - val_loss: 0.0363 - val_mean_absolute_percentage_error: 803.2571\n","Epoch 15/50\n","35/35 [==============================] - 0s 433us/sample - loss: 0.0347 - mean_absolute_percentage_error: 25.0646 - val_loss: 0.0356 - val_mean_absolute_percentage_error: 731.2593\n","Epoch 16/50\n","35/35 [==============================] - 0s 417us/sample - loss: 0.0340 - mean_absolute_percentage_error: 25.0223 - val_loss: 0.0338 - val_mean_absolute_percentage_error: 612.5067\n","Epoch 17/50\n","35/35 [==============================] - 0s 489us/sample - loss: 0.0331 - mean_absolute_percentage_error: 26.7277 - val_loss: 0.0328 - val_mean_absolute_percentage_error: 451.5154\n","Epoch 18/50\n","35/35 [==============================] - 0s 409us/sample - loss: 0.0338 - mean_absolute_percentage_error: 27.5179 - val_loss: 0.0328 - val_mean_absolute_percentage_error: 466.8643\n","Epoch 19/50\n","35/35 [==============================] - 0s 450us/sample - loss: 0.0333 - mean_absolute_percentage_error: 27.1883 - val_loss: 0.0327 - val_mean_absolute_percentage_error: 533.8986\n","Epoch 20/50\n","35/35 [==============================] - 0s 394us/sample - loss: 0.0331 - mean_absolute_percentage_error: 26.6978 - val_loss: 0.0326 - val_mean_absolute_percentage_error: 672.0956\n","Epoch 21/50\n","35/35 [==============================] - 0s 469us/sample - loss: 0.0337 - mean_absolute_percentage_error: 25.4493 - val_loss: 0.0343 - val_mean_absolute_percentage_error: 837.8459\n","Epoch 22/50\n","35/35 [==============================] - 0s 458us/sample - loss: 0.0347 - mean_absolute_percentage_error: 25.9778 - val_loss: 0.0322 - val_mean_absolute_percentage_error: 570.5813\n","Epoch 23/50\n","35/35 [==============================] - 0s 530us/sample - loss: 0.0334 - mean_absolute_percentage_error: 27.0696 - val_loss: 0.0329 - val_mean_absolute_percentage_error: 472.2430\n","Epoch 24/50\n","35/35 [==============================] - 0s 405us/sample - loss: 0.0338 - mean_absolute_percentage_error: 27.3810 - val_loss: 0.0322 - val_mean_absolute_percentage_error: 607.1464\n","Epoch 25/50\n","35/35 [==============================] - 0s 559us/sample - loss: 0.0336 - mean_absolute_percentage_error: 27.4095 - val_loss: 0.0324 - val_mean_absolute_percentage_error: 473.0745\n","Epoch 26/50\n","35/35 [==============================] - 0s 406us/sample - loss: 0.0337 - mean_absolute_percentage_error: 27.4562 - val_loss: 0.0316 - val_mean_absolute_percentage_error: 583.5190\n","Epoch 27/50\n","35/35 [==============================] - 0s 369us/sample - loss: 0.0335 - mean_absolute_percentage_error: 25.6790 - val_loss: 0.0351 - val_mean_absolute_percentage_error: 847.4083\n","Epoch 28/50\n","35/35 [==============================] - 0s 405us/sample - loss: 0.0346 - mean_absolute_percentage_error: 25.2173 - val_loss: 0.0348 - val_mean_absolute_percentage_error: 662.8919\n","Epoch 29/50\n","35/35 [==============================] - 0s 415us/sample - loss: 0.0332 - mean_absolute_percentage_error: 25.9801 - val_loss: 0.0332 - val_mean_absolute_percentage_error: 469.8535\n","Epoch 30/50\n","35/35 [==============================] - 0s 410us/sample - loss: 0.0339 - mean_absolute_percentage_error: 27.8425 - val_loss: 0.0326 - val_mean_absolute_percentage_error: 414.7518\n","Epoch 31/50\n","35/35 [==============================] - 0s 550us/sample - loss: 0.0332 - mean_absolute_percentage_error: 26.9420 - val_loss: 0.0340 - val_mean_absolute_percentage_error: 710.4662\n","Epoch 32/50\n","35/35 [==============================] - 0s 380us/sample - loss: 0.0339 - mean_absolute_percentage_error: 25.4025 - val_loss: 0.0347 - val_mean_absolute_percentage_error: 688.7251\n","Epoch 33/50\n","35/35 [==============================] - 0s 417us/sample - loss: 0.0342 - mean_absolute_percentage_error: 26.9076 - val_loss: 0.0317 - val_mean_absolute_percentage_error: 487.5617\n","Epoch 34/50\n","35/35 [==============================] - 0s 466us/sample - loss: 0.0334 - mean_absolute_percentage_error: 26.8543 - val_loss: 0.0310 - val_mean_absolute_percentage_error: 613.7100\n","Epoch 35/50\n","35/35 [==============================] - 0s 459us/sample - loss: 0.0336 - mean_absolute_percentage_error: 26.1343 - val_loss: 0.0310 - val_mean_absolute_percentage_error: 599.0154\n","Epoch 36/50\n","35/35 [==============================] - 0s 518us/sample - loss: 0.0337 - mean_absolute_percentage_error: 27.5218 - val_loss: 0.0323 - val_mean_absolute_percentage_error: 403.6833\n","Epoch 37/50\n","35/35 [==============================] - 0s 421us/sample - loss: 0.0336 - mean_absolute_percentage_error: 27.4454 - val_loss: 0.0349 - val_mean_absolute_percentage_error: 461.2988\n","Epoch 38/50\n","35/35 [==============================] - 0s 413us/sample - loss: 0.0334 - mean_absolute_percentage_error: 26.8901 - val_loss: 0.0338 - val_mean_absolute_percentage_error: 650.4165\n","Epoch 39/50\n","35/35 [==============================] - 0s 463us/sample - loss: 0.0344 - mean_absolute_percentage_error: 25.0298 - val_loss: 0.0335 - val_mean_absolute_percentage_error: 733.0479\n","Epoch 40/50\n","35/35 [==============================] - 0s 441us/sample - loss: 0.0350 - mean_absolute_percentage_error: 26.8224 - val_loss: 0.0308 - val_mean_absolute_percentage_error: 331.2224\n","Epoch 41/50\n","35/35 [==============================] - 0s 373us/sample - loss: 0.0348 - mean_absolute_percentage_error: 28.9593 - val_loss: 0.0323 - val_mean_absolute_percentage_error: 562.0246\n","Epoch 42/50\n","35/35 [==============================] - 0s 411us/sample - loss: 0.0338 - mean_absolute_percentage_error: 25.8618 - val_loss: 0.0352 - val_mean_absolute_percentage_error: 705.7799\n","Epoch 43/50\n","35/35 [==============================] - 0s 430us/sample - loss: 0.0343 - mean_absolute_percentage_error: 25.7046 - val_loss: 0.0333 - val_mean_absolute_percentage_error: 538.3356\n","Epoch 44/50\n","35/35 [==============================] - 0s 385us/sample - loss: 0.0336 - mean_absolute_percentage_error: 26.7710 - val_loss: 0.0327 - val_mean_absolute_percentage_error: 509.3722\n","Epoch 45/50\n","35/35 [==============================] - 0s 523us/sample - loss: 0.0345 - mean_absolute_percentage_error: 28.7642 - val_loss: 0.0327 - val_mean_absolute_percentage_error: 384.9824\n","Epoch 46/50\n","35/35 [==============================] - 0s 453us/sample - loss: 0.0338 - mean_absolute_percentage_error: 25.9567 - val_loss: 0.0345 - val_mean_absolute_percentage_error: 814.8223\n","Epoch 47/50\n","35/35 [==============================] - 0s 422us/sample - loss: 0.0352 - mean_absolute_percentage_error: 24.8184 - val_loss: 0.0358 - val_mean_absolute_percentage_error: 634.1329\n","Epoch 48/50\n","35/35 [==============================] - 0s 457us/sample - loss: 0.0348 - mean_absolute_percentage_error: 27.6794 - val_loss: 0.0312 - val_mean_absolute_percentage_error: 303.8320\n","Epoch 49/50\n","35/35 [==============================] - 0s 476us/sample - loss: 0.0361 - mean_absolute_percentage_error: 28.6131 - val_loss: 0.0304 - val_mean_absolute_percentage_error: 588.8770\n","Epoch 50/50\n","35/35 [==============================] - 0s 521us/sample - loss: 0.0332 - mean_absolute_percentage_error: 24.9838 - val_loss: 0.0354 - val_mean_absolute_percentage_error: 851.1025\n"],"name":"stdout"},{"output_type":"execute_result","data":{"text/plain":["<tensorflow.python.keras.callbacks.History at 0x7fa2ee09bf60>"]},"metadata":{"tags":[]},"execution_count":166}]},{"cell_type":"code","metadata":{"id":"khfs3lDztn8W","colab_type":"code","colab":{}},"source":["# Predictions\n","pred_train = model.predict(X_train)\n","pred_test = model.predict(X_test)"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"Tr2MPdIOtoIT","colab_type":"code","outputId":"63d9f4ba-c2b8-47fa-a7de-e42c2c5345bd","executionInfo":{"status":"ok","timestamp":1583941147249,"user_tz":420,"elapsed":513,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["# Train metrics\n","train_mse = tf.keras.metrics.MeanSquaredError()\n","train_mse.update_state(y_train, pred_train)\n","print(f'train_mse: {train_mse.result().numpy(): .2f}')"],"execution_count":168,"outputs":[{"output_type":"stream","text":["train_mse:  0.02\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"CL4h0blqtoER","colab_type":"code","outputId":"b828f072-6b0c-4c37-82fb-5657ff290e71","executionInfo":{"status":"ok","timestamp":1583941148049,"user_tz":420,"elapsed":576,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["# Test metrics\n","test_mse = tf.keras.metrics.MeanSquaredError()\n","test_mse.update_state(y_test, pred_test)\n","print(f'test_mse: {test_mse.result().numpy(): .2f}')"],"execution_count":169,"outputs":[{"output_type":"stream","text":["test_mse:  0.01\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"zbT59j1otfV4","colab_type":"text"},"source":["Regularization didn't make a difference in performance with this specific example because our data is generated from a perfect linear equation but for large realistic data, regularization can help our model generalize well."]},{"cell_type":"markdown","metadata":{"id":"XwSZgi_pCb3Q","colab_type":"text"},"source":["---\n","Share and discover ML projects at <a href=\"https://madewithml.com/\">Made With ML</a>.\n","\n","<div align=\"left\">\n","<a class=\"ai-header-badge\" target=\"_blank\" href=\"https://github.com/madewithml/lessons\"><img src=\"https://img.shields.io/github/stars/madewithml/lessons.svg?style=social&label=Star\"></a>&nbsp;\n","<a class=\"ai-header-badge\" target=\"_blank\" href=\"https://www.linkedin.com/company/madewithml\"><img src=\"https://img.shields.io/badge/style--5eba00.svg?label=LinkedIn&logo=linkedin&style=social\"></a>&nbsp;\n","<a class=\"ai-header-badge\" target=\"_blank\" href=\"https://twitter.com/madewithml\"><img src=\"https://img.shields.io/twitter/follow/madewithml.svg?label=Follow&style=social\"></a>\n","</div>\n","             "]}]}